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for  the  air-to-ground  combat  environment.  Both  formal  and  empirical 
models  of  decision-making  are  discussed.  Four  components  of 
decision-making:  information  selection,  prob ability  estimation,  worth 
assessment,  and  action  selection  are  identified.  Empirical  results  from 
laboratory  and  real-world  studies  are  used  to  characterize  operator 
performance  for  each  of  these  components  of  decision-making. 

'fUj  The  results  indicate  that  the  operator  is  best  characterized  in 
terms  of  empirical,  rather  than  normative,  models  of  decision-making,  in 
particular,  models  based  on  heuristic  processes  seem  to  best  characterize 
operator  performance.  Heuristic  processes  are  rules  used  by  the  operator 
which  simplify  the  decision  situation  and  result  in  predictable  errors.  £ 
Such  rules  are  used  because  of  the  operator’s  processing  limitations  and  \ 
not  necessarily  because  of  his  irrationality.  Decision-aiding  is  suggested  as 
one  way  to  overcome  these  limitations  and  improve  operator 
performance. 
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INTRODUCTION 


During  combat,  the  fighter/attack  aircrew  Is  faced  with  a  dynamic, 
complex,  and  highly  stressful  environment.  Even  In  noncombat  missions, 
the  increasing  complexity  of  avionic  systems  puts  a  high  informational 
load  on  their  operators.  The  use  of  decision  aids  is  a  possible  way  to 
reduce  the  operator’s  cognitive  load  while  improving  overall  systems 
performance.  Presently,  however,  no  overall  structure  exists  to  analyze 
the  aircrew’s  decision  environment  and  to  pinpoint  possible  areas  where 
decision  aids  might  be  useful. 

This  report  is  a  review  of  the  literature  on  human  decision-making 
with  a  focus  on  possible  applications  for  decision-aiding  on  Navy  attack 
aircraft.  The  purpose  of  the  report  is  to  isolate  general  decision-making 
characteristics  of  the  operator  for  future  comparison  with  decision 
situations  identified  as  important  for  naval  attack  missions.  The 
operator's  decision-making  characteristics  can  then  be  compared  to  ideal 
solutions  generated  by  mathematical  decision  models.  The  difference 
between  operator  performance  and  ideal  performance  will  form  one  basis  for 
developing  criteria  for  interfacing  the  operator  with  decision-aiding 
devices. 

Figure  1  shows  how  this  report  will  be  combined  with  information 
from  other  sources  in  order  to  develop  these  criteria.  Basically,  the 
criteria  will  be  based  on  three  considerations:  (1)  how  the  operator 
performs,  (2)  what  he  does,  and  (3)  what  aids  are,  or  will  be,  available. 
The  latter  two  considerations  are  subjects  of  ongoing  research. 


OVERVIEW  OF  OPERATOR  DECISION-MAKING  CHARACTERISTICS 

A  number  of  excellent  reviews  of  human  decision-making  have  been 
published  recently  (Nickerson  and  Feehrer,  1975;  Slovic  and  Lichenstein, 
1971;  Slovic,  Fischhoff,  and  Lichtenstein,  1977).  The  purpose  of  the 
present  review  is  not  to  supplant  these  reviews,  but  rather  to  outline 
the  human  operator's  decision-making  characteristics  (ODC)  in  order  to 
generate  a  detailed  scheme  of  decision-making  tasks.  The  scheme  and 
review  of  ODC  will  complement  each  other,  since  together  they  will 
characterize  both  different  decision  tasks  and  the  operator's  performance 
at  these  tasks. 


FIGURE  1.  Development  of  Criteria  for  Decision-Aiding 
in  Fighter/Attack  Aircraft. 

Figure  2  depicts  a  simplified  model  of  probabilistic  decision-making. 
It  shows  four  stages  of  human  decision-making:  (1)  information  selection, 
(2)  probability  estimation,  (3)  worth  assessment,  and  (4)  action  selec¬ 
tion.  The  first  stage  consists  of  generating  hypotheses  concerning  states 
of  the  world  and  selectively  choosing  data  (dj. . .dj . . .dn)  in  the  environ¬ 
ment  to  test  the  hypotheses.  Stage  two  involves  probability  estimation 
of  different  states  of  the  world  based  on  the  data  [?(%  |  d^) . .  .P(Hn  |  d^)  ] . 
Stage  three  involves  generating  possible  outcomes  (4»^...4>n)  and  evaluating 
their  worth  (<{>1  <  <j>j  <  ...  <  4>n) .  The  final  stage  is  the  consideration 
of  alternative  actions  in  light  of  the  information  from  stages  two  and 
three. 

An  example  may  make  the  theoretical  issues  involved  in  decision¬ 
making  more  concrete.  An  attack  pilot  must  decide  whether  to  launch  a 
missile  at  a  ship  for  which  he  has  only  a  noisy  image  on  his  CRT  scope. 

His  first  task  is  to  decide  how  much  information  to  collect  before 
deciding  whether  to  launch  the  missile  or  not.  Too  much  information  will 
result  in  putting  himself  in  danger  of  being  shot  down  and  too  little 
might  result  in  his  making  the  wrong  decision.  The  operator’s  information 
selection  characteristics  will  be  discussed  as  an  iterative  process 
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(c.f.,  the  feedback  loop  In  Figure  2),  since  the  operatur  must  know  his 
state  of  uncertainty  and  the  consequences  of  his  possible  actions  before 
he  can  decide  how  much  information  to  select.  However,  it  is  also  the 
logical  first  step  in  decision-making  since  all  the  other  stages  require 
sufficient  Information  in  order  to  make  even  preliminary  estimates. 

After  gathering  information,  the  pilot  can  begin  to  form  hypotheses 
about  what  he  is  viewing  on  his  display.  Based  on  both  the  data  (dj) 
and  his  prior  experience,  the  pilot  generates  likely  hypotheses. 

In  our  example,  the  imagery  of  the  ship  outline  on  his  scope,  especially 
the  two  faint  superstructures,  suggests  an  enemy  destroyer  or  a  friendly 
combatant  (Hf  or  He) .  There  is  also  a  slight  possibility  that  the  image 
is  a  friendly  hospital  ship  (Hh).  Given  the  data  (dj)  and  anything  else 
the  operator  knows  about  the  situation,  the  probability  estimation  tasks 
consist  of  quantifying  the  operator's  subjective  level  of  uncertainty 
concerning  each  of  the  hypotheses  [i.e.,  P(He|dj);  P(tic|dj)  or  P(Hh|dj)J. 

However,  his  final  decision  depends  not  only  on  his  subjective 
probability  estimates,  but  also  on  his  estimate  of  the  consequences  of 
his  actions.  The  third  stage  of  decision-making  involves  assessing  the 
consequences  of  possible  outcomes  and  is  referred  to  as  "worth  assessment." 
In  our  example,  the  pilot  must  evaluate  the  possible  outcomes  of  sinking 
or  not  sinking  the  ship  imaged  on  his  CRT.  For  example,  the  benefits  of 
sinking  an  enemy  ship  [i.e.,  i£e]  might  not  balance  the  risk  of  sinking 
a  friendly  hospital  [ip},]  or  combat  [<fcj  ship.  The  consequences  of  not 
sinking  these  ships  must  also  be  considered  [e.g.  ,  4>e  1  •  The 

discussion  of  worth  assessment  will  -oncentrate  on  techniques  for  eval¬ 
uating  consequences  and  the  performance  charac ter is  tics  of  the  operator 
using  these  techniques. 

The  final  stage  of  decision-making  consists  of  generating  decision 
rules  to  select  the  best  course  of  action  using  both  the  previously 
estimated  probability  and  worth  information.  For  example,  because 
sinking  a  friendly  may  be  much  worse  than  to  miss  sinking  an  enemy  ship, 
the  operator  may  have  a  90%  criterion  in  order  to  launch  the  missile. 

If  he  is  90%  sure  that  it  is  an  enemy  ship,  he  launches  the  missile; 
otherwise,  he  continues  to  track  the  ship.  Thus  the  output  in  Figure  2 
is  a  result  of  both  the  consequences  of  the  action  and  the  operator’s 
state  of  subjective  uncertainty  as  well  as  the  particular  decision  rule 
he  chooses. 

This  example  will  be  used  throughout  the  paper  to  introduce  the  dif¬ 
ferent  stages  of  decision-making.  It  would  be  misleading  to  assume  that 
the  example  captures  the  difficulties  inherent  in  real-world  decision¬ 
making.  In  particular,  the  complexities  associated  with  structuring  the 
decision  situation  are,  for  the  most  part,  unexplored.  This  is  because 
little  research  has  been  done  concerning  how  the  operator  structures  his 
decision  environment  (e.g.,  deciding  what  options  are  realistic,  what 
outcomes  are  possible...).  Rather,  the  purpose  of  the  example  is  to 
Illustrate  the  issues  relating  to  abstract  decision  models  with  an  easily 
understood,  concrete  example. 
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The  four-stage  model  of  decision-making  (Figure  2)  will  be  the 
general  framework  for  discussing  features  of  various  decision  tasks 
and  for  characterizing  the  operator's  performance.  The  functions  in 
the  boxes  from  Figure  2  can  be  interpreted  in  two  ways.  One,  they  can 
be  normative  functions  following  well-defined  mathematical  rules  to 
optimize  the  expected  outcome.  Two,  they  can  be  descriptive  functions 
characterizing  what  the  operator  actually  does.  Both  interpretations 
are  important  in  order  to  optimize  performance  in  decision-making 
situations  when  a  human  operator  is  involved. 

Figure  2  itself  is  too  simplified,  since  it  describes  a  static 
decision  situation  and  most  decisions  are  made  in  dynamic  environments 
(cf.  Rapaport,  1975).  However,  it  does  describe  the  main  con^onents  of 
the  decision-making  process  and  seems  to  catch  the  essence  of  other 
decision-making  models  (Nickerson  and  Feehrer,  1973)  while  serving  as 
a  convenient  framework  for  this  report. 

Probability  estimation  will  be  discussed  first  since  this  seems  to 
be  a  necessary  first  step  in  decision-making  for  uncertain  situations. 
Information  selection  will  be  discussed  last  since  it  involves  concepts 
which  will  be  developed  in  discussing  other  stages  of  decision-making. 


PROBABILITY  ESTIMATION 


OVERVIEW 

Figure  3  illustrates  the  general  framework  of  the  probability  esti¬ 
mation  task.  The  operator  must  structure  the  problem  by  enumerating 
possible  real-world  states  that  are  referred  to  as  hypotheses  (Hi. . .H^. . .Hn) 
His  task  is  to  express  his  degree  of  uncertainty  regarding  each  H^,  based 
on  its  prior  probability  distribution  (P(Hk))  and  its  relationship  to 
Incoming  dat/i  (di...dj).  The  quantification  of  the  operator's  subjective 
degree  of  uncertainty  is  his  probability  estimate. 

For  the  example,  the  pilot  would  have  two  sources  of  information: 

(1)  his  subjective  uncertainty  concerning  the  image  on  the  screen,  and 

(2)  a  preflight  intelligence  briefing  which  indicated  there  was  a  75% 
probability  of  an  enemy  ship  being  in  the  area  and  a  20%  probability  that 
a  friendly  destroyer  might  be  in  the  area.  The  briefing  also  reported  a 
remote  possibility  that  a  friendly  hospital  ship  strayed  off  course  and 
was  in  the  area.  The  pilot's  estimation  task  is  to  quantify  the  proba¬ 
bilities  for  each  of  the  possible  hypotheses. 
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Uutort  uuate Iv  ,  there  4tt'  scuiaut  ic  and  pin  K>si'plu,  di  1 1  ereuce's  ot 
opinion  concerning  the  definition  ot  probahil.tv.  Presently,  tlieie  aic 
two  main  definitions:  (11  the  sub  Joel  i  ve  for  personal)  Jot  nit !  ion  .uni 
(-')  tin;  frequency  (or  object  Ive)  definition  (lioel,  t*J-‘l;  Lee,  I'i.’P. 

The  subjective  definition  (deKlnet t i ,  lvi.’0;  Savage,  states  that 

probability  is  the  degree  of  uncertainty  expressed  behavioral lv  which 
follows  certain  mat  hemal  leal  axioms  fe.g.,  the  quantities  sum  to  1, 
are  not  negative,  etc. I,  The  frequency  definition  uses  (lie  same  axioms, 
but  defines  probability  as  the  expected  distribution  lot  epoated 
out  co  toes  ot  the  events  being  assigned  probabilities.  Ihe  tain  ptoblem 
with  t  lit'  tirst  definition  Is  that  the  ptobabtlily  ot  an  e  nt  can  ditto! 
trow  operator  to  operator,  whereas  the  second  del  ini l ion  is  not  rich 
enough  to  allow  probability  estimates  in  a  great  number  ot  cases  le.g., 
probability  ot  a  Russian  attack  tomorrow)  ffdvards,  1  t  ndmatt  and  Savage', 
l ) ;  Savage  ,  l "> , 

Pin,  a(> 

Ptw.laj 
^iHkKV 

l*.H  ,  .jvl,' 

riH„uv 


WN 6 Hit  i  dj'  IS  T Hfe  PS08ASU.il'>  or  m'iSOTHISiS  e  uIVtN  OAU'M  i 

I'liUiRK  I.  Outline  of  Probability  Kst  irnat  ion  Task. 

I'he  problem  becomes  even  murkier  because'  the-  operator's  probabilits 
estimates  elo  not  necessarily  tol  low  the  axioms  that  define  probability 
and  t  he  re  t  o  re  are  not  subjective  probab  i  l  1 1  ies .  In  this  tvpoit,  the' 
te>rm  subjective'  probability  will  be  assume'd  to  re'ptese'nt  ideal  operator 
per to  nuance  and  mav  or  may  not  describe  actual  operator  pertovmanoe. 
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NORMATIVE  MODELS  OF  ODC 


B ayes  Theory m 

The  traditional  approach  for  investigating  the  operator's  proba¬ 
bility  estimation  characteristics  was  to  compare  his  performance  to 
'ideal'  mathematical  models  such  as  Bayes  Theorem  (Equation  1) 
(Peterson  and  Beach,  1967) 


where 


P(Hk)  P(dj | Hk) 
n 

T  (n.)  P(djn.) 
i=  l 


(i) 


P(H|t|dj)  =  probability  of  hypothesis  k  given  datum  j 
(i.e.,  posterior  probability) 


P(Hk) 

»’(d,|Hk) 

n 

L  (H.)  P(d  |H  ) 
i»l  J  1 


=  prior  probability  of  hypothesis  k 

=  probability  of  sampling  datum  j  given 
hypothesis  k  is  true 

*  probability  of  the  data  summed  over  all  the 
possible  hypotheses  ()’  P(1L)  =  1) 


Most  traditional  research  involves  binomial  data,  and  it  is  convenient 
to  compare  the  relative  probability  of  two  hypotheses  (e.g.,  and  H^) 
as  an  odds  ratio. 


P(H  jd  )  P(Hk)  P(d  |Hr) 

. » . . —  =  — ■ — ■ — —  •  - - or 

P(H1 |d  . )  P(HX)  P(d  ^ [Hx)  ’ 

I!  *=  •  LK 

k-1  o 
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where 

f2  =  posterior  odds  of  hypotheses  k  to  1 
k-1 

*  prior  odds 
o 

LK  =  likelihood  ratio 

Equation  3  shows  the  multiplicative  effect  of  sampling  n  samples  of 
conditionally  independent  data 

n 

n  =  n  n  lk  o) 

k-l  o 

The  revision  of  probability  estimates  as  more  data  are  sampled  is 
referred  to  as  Bayesian  updating. 

If  we  posit  that  the  pilot  in  our  example  is  a  Bayesian  estimator, 
his  first  task  is  to  estimate  his  subjective  probability  of  the  partic¬ 
ular  image  on  his  display,  given  the  three  hypotheses  [e.g.,  P(dj | H f ) 

=  .80;  P(dj | Hf )  =  .19;  P(dj)Hh)  =  .01],  At  this  point,  the  operator 
may  narrow  his  hypotheses  to  either  a  friendly  or  enemy  ship,  since  the 
hospital  hypothesis  seems  extremely  unlikely.  The  Bayesian  operator 
combines  these  estimates  with  the  intelligence  information  [P(Hf)  =  .73; 
P(He)  =  .20]  in  order  to  compute  the  odds  of  the  ship  being  enemy 
using  Equation  2, 


a 

e-f 


Two  points  should  be  made  concerning  the  above  computation.  (1)  The 
operator  state  of  uncertainty  is  high  and  thus  his  decision  must  rest 
mostly  on  the  expected  consequences  of  his  actions.  (2)  His  prior  odds 
and  likelihood  information  seem  to  cancel  each  other  out.  The  latter 
point  is  important,  since  evidence  will  be  introduced  later  which  suggests 
that  the  operator  tends  to  ignore  the  implications  of  prior  odds. 

At  first  glance,  it  seems  a  bit  silly  to  argue  that  the  operator 
uses  Bayes  theorem  in  a  combat  environment.  However,  this  does  serve 
as  a  useful  first  approximation  model  of  the  operator's  probability 
estimation  performance.  By  comparing  the  Bayesian  estimate  to  the 
operator's  actual  estimate,  it  should  be  possible  to  see  how  well  this 
model  predicts  operator  performance.  The  initial  research  in  this  area 
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concluded  that  Che  operator  was  a  good.  If  imperfect,  "Bayesian"  estimator 
(Peterson  and  Beach,  1967).  It  seemed  as  if  the  operator  was  estimating 
using  something  very  like  Bayes  theorem,  but  due  to  his  processing 
limitations  was  underestimating  probability. 

In  order  to  understand  this  research,  a  sort  of  "generic"  experimental 
paradigm  is  discussed  next.  Most  of  the  research  reviewed  in  this  section 
(on  normative  models)  used  experimental  paradigms  very  similar  to  the 
example  to  be  presented,  and  one  of  the  limitations  to  the  research  is  the 
possible  "artificiality"  of  these  paradigms.  The  example  will  also  help 
to  illustrate  the  mechanics  of  the  Bayesian  updating  task  since  the  norma¬ 
tive  solution  is  Equation  3. 

There  are  two  urns  (Hi  and  H2)  from  which  the  data  (dj)  can  be 
drawn.  Urn  1  has  70%  red  balls  (dr)  and  30%  blue  balls  (db) ;  the  data 
proportions  are  the  opposite  for  Urn  2.  The  experimenter  chooses  one 
urn  at  random  (prior  probability  ■  .5)  and  draws  12  balls,  one  at  a 
time,  from  the  urn  (replacing  each  ball  before  drawing  another) . 

Given  the  fact  that  eight  red  balls  and  four  blue  balls  were  drawn, 
the  subjects  nust  estimate  the  probability  (or  odds)  that  the  balls 
came  from  Urn  1.  The  subjects'  estimated  odds  are  almost  always  too 
conservative  (i.e.,  they  tend  to  underestimate  the  odds)  for  this  type 
of  task  wherein  the  Bayesian  solution  (Equation  3)  yields  odds  of  30  to  1 
favoring  hypothesis  1. 

_  U5)  .  (0.78  0.33)  _  (OjjY*  _  __ 

1‘2  <•»"  <0.74  0.38)  '  '<W  ' 

Figure  4  shows  a  conservative  operator's  probability  estimates 
compared  to  the  optimal  solution  and  an  excessive  operator.  The  con¬ 
servative  operator  underestimates  the  impact  of  data,  whereas  the 
excessive  operator  overvalues  their  impact.  Notice  that  the  more 
extreme  the  actual  probabilities  (abscissa) ,  the  more  hypothetical 
operators  deviate  from  the  normative  model.  In  general,  it  was  found 
that  the  more  diagnostic  the  data  (i.e.,  the  degree  to  which  the  data 
imply  a  particular  hypothesis)  the  more  subjects  deviated  from  the 
Bayesian  model  (Peterson  and  Beach,  1967;  Slovic  and  Lichenstein,  1971). 

The  almost  universal  finding  of  conservatism  when  using  paradigms 
similar  to  our  example  was  taken  as  evidence  that  the  normative  approach 
is  fruitful.  The  operator  apparently  used  some  Bayesian  method  to 
update  his  probability  estimates,  but  underestimated  their  magnitude. 

A  correction  for  the  slope  values  in  Figure  4  should  result  in  a  con¬ 
servative  operator  computing  optimal  solutions.  Most  of  the  normative 
research  attempted  to  find  the  reason  for  suboptimal  performance. 
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FIGURE  4.  Hypothetical  Characteristics  of  Operator  in 
Bayesian  Updating  Task. 
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Empirical  Results 

The  two  main  hypotheses  for  suboptima]  performance  were  misper¬ 
ception  and  misaggregation.  Misperception  refers  to  the  tendency  of 
the  operator  to  misperceive  the  properties  of  the  underlying  probability 
distribution.  Peterson,  Ducharme  and  Edwards  (1968)  found  that  subjects 
generated  probability  distributions  that  were  too  flat.  If  the  proba¬ 
bility  estimates  from  these  flat  distributions  were  used  with  Equation  3, 
their  subjects'  likelihood  ratios  could  be  predicted. 

Misaggregation  refers  to  the  inability  of  the  operator  to  aggregate 
probability  estimates  optimally  over  a  number  of  updating  trials  as 
implied  by  Equation  3.  Ducharme  and  Peterson  (1968)  found  that  subjects' 
estimates  of  P(dj|lig)  for  a  single  trial  were  nearly  optimal  but  they 
could  not  properly  aggregate  the  estimates  over  a  number  of  trials. 
Apparently  operators  both  misaggregate  and  misperceive  the  import  of 
data  in  the  updating  task  (Peterson  and  Beach,  1967;  Slovic  and 
Lichtenstein,  1971). 

Besides  misperceiving  and  misaggregatlng,  the  operator  has  other 
suboptlmal  estimation  characteristics.  Operators  have  a  bias  against 
giving  extreme  probability  responses  (Ducharme,  1970).  Also,  equal 
likelihood  ratios  (LK)  are  not  interpreted  as  equal  (e.g.,  0.50/0.25  i 
0.04/0.02;  Beach,  1968). 

Furthermore,  fallacious  dependencies  are  seen  to  exist  among 
Independent  events  which  cause  operators  to  underestimate  aggregated 
probabilities  (Brickman  and  Pierce,  1972;  Kahneman  and  Tversky,  1972). 

For  example,  Brickman  and  Pierce  (1972)  told  their  subjects  the  P(dj|li^) 
for  a  single  trial  in  an  experiment  similar  to  the  Urn  example.  The 
subjects  were  told  the  color  of  balls  drawn  from  Urn  k  for  the  past  n 
trials  and  told  to  estimate  the  probability  of  a  particular  color  ball 
being  drawn  on  trial  n  +  1.  If  subjects  were  told  that  the  preceding 
four  balls  were  the  same  color,  they  underestimated  P(d]|llk)  on  trial 
five,  because  they  assumed  that  drawing  five  balls  of  the  same  color 
was  extremely  unlikely.  This  tendency  is  well  known  in  betting  as 
"gambler's  fallacy"  (Lee,  1971). 

Winkler  and  Murphy  (1973)  suggest  that  conservatism  may  be  an 
artifact  of  the  experimental  paradigms  used.  These  paradigms  are 
highly  artificial,  whereas  data  in  the  real  world  are  noisy  and  redun¬ 
dant  (i.e,,  not  Independent),  making  it  naturally  nondiagnostic.  If 
subjects  treated  data  (non-noisy  and  independent)  in  these  experiments 
like  real-world  data,  they  would  be  labeled  conservative. 
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The  manner  in  which  probability  estimates  are  made  influences  the 
subject's  degree  of  conservatism.  Stated  probability  estimates  were 
more  conservative  than  were  probabilities  inferred  from  overt  acts 
(Codden,  1976).  Since  subjective  probabilities  technically  are 
measured  In  terms  of  overt  acts  (such  as  betting,  explicit  preferences, 
etc.)  the  suboptlmal  characterlst ics  may  be  partially  an  artifact  of 
response  mode  (deFinettl,  1970). 

Titus,  there  seems  to  be  some  doubt  as  to  whether  conservatism  is 
the  result  of  ODC  or  simply  of  the  artificial  paradigms  used.  Recent 
research,  using  botii  real-worid  data  and  different  research  paradigms, 
has  found  that  operators  tend  to  be  excessive  in  as  many  situations  as 
those  in  which  they  tend  to  be  conservative  (Fischhoff,  Slovic,  and 
Lichtenstein,  1977;  Lichtenstein,  Fischhoff,  and  Phillips,  1977;  Pit/, 

1975;  Schaeffer,  1977;  Slovic,  1972;  Slovic,  Fischhoff,  and  Lichtenstein, 
1977).  These  later  findings  undermine  the  whole  concept  of  the  human 
operator  being  a  suboptlmal  Bayesian  performer.  To  quote  Kahneman  and 
Tversky  (1972),  the  operator,  "in  his  evaluation  of  evidence,  is  apparently 
not  a  conservative  Bayesian;  he  is  not  Bayesian  at  all." 

There  has  been  a  general  shift  In  approach  since  L972  from  modeling 
the  operator  as  Bayesian  to  attempting  to  find  appropriate  descriptive 
models  for  the  ODC  (Slovic,  Fischhoff,  and  Lichtenstein,  1977). 


DESCRIPTIVE  MODELS  OF  ODC 

Operator  Bjtas  and  Heuristic  Techniques 

Most  descriptive  models  Imply  that  the  human  operator  uses  heuristic 
(l.e.,  simplifying)  assumptions  to  generate  probability  estimates.  A 
heuristic  is  a  rule  that  simplifies  computation,  but  does  not  always 
result  In  the  correct  conclusion.  It  may  replace  formal  models  (such 
as  Equations  1,  2,  and  3)  and  lead  to  predictable  errors  (biases). 
Different  heuristics  are  used  for  different  tasks.  Tversky  and 
Kahneman  (1975)  (also  Kahneman  and  Tversky,  1972;  Kahneman  and  Tverskv, 
1973)  have  identified  the  following  three  heuristics: 

1.  Representativeness:  The  use  of  salient  properties  of  the 
parent  population  such  as  proportions,  modal  responses,  etc.,  to  predict 
sample  outcomes.  This  heuristic  Involves  the  following  biases: 

Ignoring  the  implications  of  prior  odds,  sample  size,  rundomness,  and 
unreliability,  and  overweight Ing  other  features  of  the  hypothetical 
population.  This  type  of  heuristic  is  almost  deterministic.  Some 
examples  are  the  beltef  that  long-haired  students  are  radical,  and 
that  good  law  students  necessarily  become  snccesslul  lawyers.  Observed 
proportions  with  a  small  sample  size  are  held  to  be  as  valid  as  same 
proportions  with  larger  sample  size. 
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2.  Availability;  The  generation  of  frequency  distributions  by 
recalling  examples  of  the  relevant  events.  This  heuristic  Involves 
biases  associated  with  memory  search.  For  example,  vivid  possibilities 
are  judged  more  probable  than  less  vivid  ones. 

3.  Adjustment  and  Anchoring:  A  point  in  the  distribution  is  used 
as  a  start  and  the  distribution  is  adjusted  to  this  anchor.  Bias  for 
this  heuristic  is  related  to  the  context  of  the  available  information. 
If  subjects  are  given  extreme  Information,  their  distribution  will 

be  adjusted  to  this  information  differently  than  if  they  are  given 
median  characteristics  of  the  distribution.  For  example,  subjects 
will  construct  different  city  population  distributions  if  they  are 
told  that  95%  of  the  cities  in  Brazil  have  a  population  under  25,000 
than  if  they  are  told  that  the  median  city  in  Brazil  has  5,000  people. 

Slovic  (1972)  found  that  a  subject's  probability  estimates  depend 
on  such  factors  as  response  compatibility  and  concreteness  of  the 
stimulus  situation.  The  basic  import  of  heuristic  models  is  that  the 
operator's  strategy  is  more  holistic  than  analytical.  The  operator 
will  attempt  to  use  any  available  information  to  improvise  a  simple 
strategy  in  order  to  make  estimates. 

A  good  example  comes  from  research  done  by  Marks  and  Clarkson 
(1972,  1973).  For  the  Urn  example  given  in  the  previous  section,  it 
can  be  shown  that,  since  P(dg|Hi)  *=  1  -  P(dg| H2) ,  Equation  3  can  be 
rewritten  as; 


£2  *  £2  CKr  k  (4) 

1-k  o 


where 


r  -  number  of  red  balls 
b  »  number  of  blue  balls 

Although  it  simplifies  computation,  this  relationship  is  not  obvious. 
It  Implies  that  the  posterior  odds  would  be  the  same  for  samples  of 
four  red  and  one  blue  ball  as  for  54  red  and  51  blue  balls.  This 
apparently  is  counter-intuitive,  because  performance  of  subjects  in 
Marks  and  Clarkson's  experiments  (1972,  1973)  was  best  predicted  by 
Equation  5. 


J2  -  S2l 
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Although  Equation  5  has  no  basis  In  probability  theory,  40*  ot  the 
subjects  reported  using  it.  Those  subjects  were  using  a  simple  strategy 
that  takes  advantage  ot  representative  Information  to  est l mate  proba- 
bl  It  ties. 

Statistics  is  a  difficult  and  not  always  intuitive  discipline. 
Tverskv  and  Kahnemann  (197S)  point  out  that  the  intuition  of  even 
those  researchers  with  years  of  experience  in  making  statistical 
Inferences  often  runs  counter  to  normative  prescriptions.  It  is  not 
surprising,  therefore,  that  the  operator's  intuition  is  often  at  odds 
with  normative  models. 


Kmjp  l  r  l  ea  l  Res  ear  c  h 

Most  research  dealing  with  operator  biases  and  heuristics  tend  to 
concentrate  on  what  psychological  strategies  the  subject  actually  employs 
rather  than  to  posit  some  subopt imal  Bayesian  process.  Bayes  theorem 
is  a  yardstick  rather  than  a  model  of  human  performance  lor  this  type 
of  approach. 

Bayes  theorem  states  that  odds  generated  t rom  the  data  must  be 
weighted  by  the  prior  odds  in  order  to  compute  the  present  probability. 

A  good  deal  of  research  Indicates  that  subjects  ignore  prior  odds  for  a 
number  of  different  experimental  paradigms  (Bar-Ill  l  lei ,  ll)7S;  l.yon  and 
Slovlc,  N7b;  Nlsbett  and  Borgida,  ld7S;  Tverskv  and  Kahuemau,  1^7/1. 

The  operator  does  not  always  understand  how  base  rates  (i.e,,  prtot 
odds )  attect  probability  estimates.  For  example,  subjects  were  told 
that  the  base  rate  lor  cabs  in  a  city  was  blue  and  1  v,  green.  Their 
task  was  to  assess  the  probability  that  a  witness  who  could  tell  blue 
from  green  cabs  801  of  the  time  actually  saw  a  green  cab.  Few  subjects 
considered  the  effect  of  the  different  base  frequency  for  blue  and  green 
cabs  result tng  in  a  modal  response  ot  801  (which  is  the  witness’  hit  rat  el 
whereas  the  Bayesian  solution  was  4l'i.  Changes  in  the  base  rate,  cover 
story  or  hit  rate  tailed  to  change  these  findings  (l.yon  and  Slovlc,  l'>/;t. 

Tversky  and  Kahuemau  (ld/7>  argue  that  the  operator  ignores  base 
rate  data  when  other  data  such  as  hit  rate  are  seen  to  cause  the  phenomena 
being  estimated.  Thus,  the  subjects  in  l.von  and  Slovlc's  experiment 
thought  that  the  witnesses'  hit  rate  was  the  cause  of  report  tug  the 
correct  cab  color  and  felt  the  relative  frequency  ot  the  cabs  was 
unimportant.  Maya  Bar-Ill  1  lei  (1^77)  changed  the  stow  ot  the  cabs  so 
that  hit  rate  was  replaced  by  a  random  process  (although  in  Bayesian 
terms  the  solut Ion  was  the  Name).  In  this  case,  because  there  was  no 
causal  link  between  hit  rate  and  the  subject's  report,  many  ot  the 
subjects  considered  base  rate  when  making  their  estimates.  The  operator 
Interprets  the  validity  of  different  data  sources  In  terms  of  theit 
psychological  significance  rather  In  terms  ot  their  normative  or  Bayesian 
s  tgui  f  leance.  (Har-Hillel,  ld77;  Tversky  and  Kuhneraan,  l‘*77). 
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The  operator  is  also  influenced  by  know i ng  the  outcome  when  he 
assesses  what  his  prior  probabilities  would  have  been.  This  is  a  l.iirlv 
general  phenomenon  influencing  both  the  operator’s  probability  estimates 
and  his  memory  of  these  estimates  (Fischhoff,  1975,  1977).  This  "know 
it  all  along"  attitude  causes  the  operator  to  underestiwat e  his  previous 
uncertainty  once  the  true  state  of  the  world  is  revealed  to  him. 
Furthermore,  the  operator  tends  to  be  excessively  confident  concerning 
his  knowledge  of  the  world.  Pitz  (1975)  found  his  subjects'  confidence 
limits  far  too  narrow  when  Judging  the  populations  of  different  cities. 
Fischhoff,  Slovic  and  Lichtenstein  (1977),  using  general  knowledge 
questions,  found  their  subjects'  odds  estimates  to  be  extremely  excessive. 
Lichtenstein  and  Fischhoff  (1977)  found  their  subjects  were  only  moder¬ 
ately  overconfident,  but  degree  of  overconfidence  was  not  significantly 
decreased  by  comparing  expert  and  non-expert  subjects. 

Shanteau  (1972)  using  "functional  measurement"  techniques  found 
evidence  that  his  subjects  were  vising  an  additive  averaging  rule  and 
not  the  multiplicative  Bayesian  rule  when  updating  their  probability 
estimates.  Kils,  Seaver  and  Edwards  (1977)  found  that,  allowing  the 
subjects  to  update  using  a  logarithmic  response  scale  caused  their 
subjects  estimation  performance  to  improve.  Since  Bayesian  updating  is 
additive  on  a  logarithmic  scale,  this  further  suggests  that  some  non- 
optimal  additive  averaging  rule  may  be  used  as  a  heuristic  for  the 
updating  task. 

The  research,  taken  together,  belies  the  usefulness  of  assuming  the 
operator  is  a  Bayesian  probability  estimator.  Rather,  ODC  seems  to  In' 
task-specific.  If  the  operator  is  asked  to  construct  confidence  limits 
on  information  about  which  he  knows  little,  tile  limits  will  be  excessive 
(Fischhoff,  Slovic,  and  Lichtenstein,  1977;  Lichtenstein,  Fischhoff,  and 
Phillips,  1977;  Pitz,  1975).  However,  for  tasks  in  which  he  must  update 
probability  Information,  the  operator  will  usually  be  conservative 
(Peterson  and  Beach,  1967).  The  tasks  reviewed  so  far  involved  either 
updating  odds  as  new  information  is  received  or  estimating  the  probability 
of  an  event  on  the  basis  of  what  the  operator  knows.  The  next  seel  ions 
Involve  more  complex  tasks  and  the  final  section  reviews  what  we  know 
about  operator  performance  in  real-world  tasks. 
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COMPLEX  ESTIMATION  TASKS 


Multistage  Decisions 

Multistage  inferences  involve  data  states  that  are  removed  from  the 
hypotheses  by  one  or  more  stages.  The  operator  knows  the  relationship 
of  a  data  set  (dj. . .dj . . .dn)  to  the  hypothesis  set  (i.e.,  P(dj|H^)  is 
known).  The  operator  also  knows  the  relationship  of  a  second  data 
source  (ri. . .rg. . .rn)  to  the  data  set  [e.g.,  if  barometer  1  indicates 
30  (r^),  the  probability  that  the  atmospheric  pressure  is  actually  30 
is  0.97  (i.e.,  P(djJri)].  The  operator's  task  is  to  infer  the  posterior 
odds  of  Hit,  if  be  is  given  ri«  The  optimal  solution  is  given  by  Equation  6. 


ft 

k-l 


EPQrjH,)  P(dj|Hk) 

P(dj  |HX)  o 


(6) 


where  P(r*)  and  P( d j ( H^)  are  conditionally  Independent  (Schum  and 
Pfeiffer,  1973). 

Some  examples  of  multistage  inference  problems  are:  attempting 
to  compute  the  probability  of  real-world  states  (H. . .H^. . .Hn)  if 
sensor  reports  (rj. . . ri. . .rn)  are  unreliable  indicators  of  data  states 
(dj_. .  .dj . .  .dn) ,  reports  from  spies  with  differing  credibilities,  or  a 
composite  probability  from  witnesses  who  saw  a  crime. 

Equation  7  is  useful  for  multistage  inference  concerning  two  possible 
data  states  if  only  a  general  indication  of  the  reliability  (W)  of  the 
data  source  is  known  (Snapper  and  Fryback,  1971). 


ft 

k-l 


P(dj|Hk)  +  p 
P(dj)H1)  +  p 


ft 


(7) 


where  p 


1-W 

2W-1 


and  is  undefined  when  W 


.5. 
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Since  neither  equation  is  particularly  intuitive,  it  is  not 
surprising  that  subjects  use  heuristics  in  this  task  (Gettys,  Kelly, 
and  Peterson,  1973;  Johnson  et  al,  1973;  Snapper  and  Fryback,  1971). 

The  "as-if"  heuristic  involves  ignoring  the  probabilities  of  the  initial 
data  source  (r^)  after  choosing  the  most  probable  source.  For  example, 
a  spy  reports  that  enemy  ships  are  leaving  the  port;  this  datum  would 
be  used  to  assess  the  probability  of  attack.  The  probability  that  the 
enemy  ships  did  not  leave  port  (i.e.,  spy  is  lying)  is  not  considered. 
The  operator  acts  "as-if"  the  report  is  perfectly  reliable.  This, 
of  course,  makes  the  estimates  excessive. 

For  the  "best  guess"  heuristic  (Equation  8),  the  operator  considers 
only  the  most  probable  report  and  weights  the  posterior  odds  with  the 
reliability  of  this  report  (W|). 

P- =  W.  LK  ^  (8) 

k-1  10 


Equation  8  seems  to  predict  operator  performance  best  for  a  variety  of 
multistage  scenarios  (Gettys,  Kelly,  and  Peterson,  1973;  Johnson  et  al, 
1973;  Snapper  and  Fryback,  1971). 

There  is  evidence  that  the  operator  treats  nonstationary  probability 
situations  like  a  multistage  inference  task.  In  nonstationary  situations, 
the  probability  of  a  datum,  given  an  hypothesis,  is  variable.  For 
example,  the  probability  that  enemy  ships  leaving  port  implies  an  attack 
can  change  with  a  change  in  political  climate.  Operators  apparently  can 
use  information  that  the  underlying  probability  distribution  is  subject 
to  change  in  the  same  manner  as  they  use  other  multistage  information 
(Chinnis  and  Peterson,  1968;  1970). 
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Conditional  Non-lade pendence 

Equation  3  Implies  that  the  data  being  sampled  are  conditionally 
independent,  i.e.,  P(d1,  d2|Hj<)  =  P(di|Hk)  P(d2|Hk).  In  the  real  world, 
some  data  are  redundant  [P(di  n  d2)  >  Pdi  Pd2],  and  some  are  particularly 
informative  (P(di  n  d2>  <  Pdi  Pd2J. 

Lichtenstein  (1972)  found  Equation  3  to  be  a  robust  predictor  of 
disease  type  even  when  the  data  had  considerable  conditional  non¬ 
independence.  Since  the  type  of  non-independence  was  not  discussed  in 
detail,  it  is  difficult  to  generalize  these  results  to  other  situations. 

Edwards  (Beach,  1975;  Domas  and  Peterson,  1972)  suggests  using  a 
decision-aiding  system  he  named  PIP:  the  operator  makes  estimates  of 
P(dj|Hjt)  and  computers  update  these  estimates  using  Equation  3.  In 
general,  this  improves  the  efficiency  of  the  updating  task  (Domas  and 
Peterson,  1972;  Schaefer,  1972).  However,  when  the  data  presented  to 
their  subjects  were  redundant,  Domas  and  Peterson  (1972)  found  that  PIP 
was  less  accurate  than  the  posterior  odds  aggregated  by  their  subjects. 
This  supports  the  contention  (Winkler  and  Murphy,  1973)  that  conservatism 
for  the  updating  task  may  reflect  the  operator's  experience  with  real- 
world  data  which  are  often  redundant.  There  is  evidence  that  the  oper¬ 
ators  do  take  redundancy  into  account  as  well  as  make  allowances  for  data 
which  they  feel  are  particularly  informative  (Wyer,  1970).  However, 
considering  the  numbet  of  biases  subjects  show  in  the  updating  task,  it 
seems  unlikely  that  their  estimates  are  more  realistic  than  Bayesian 
ones.  More  information  is  needed  on  exactly  how  operators  aggregate 
non-independent  data.  It  seems  probable  that  they  use  a  simple  heuristic 
rather  than  attempt  to  compute  the  often  complex  dependencies  that  can 
exist  among  data  sources. 

A  simple  optimal  solution  is  given  by  Zlotnick  (1968).  He  suggests 
combining  different  data  sources  into  bundles  which  are  themselves 
conditionally  independent.  Then,  the  operator  (or  a  computer)  can 
aggregate  over  bundles  using  Equation  3. 


Probabilistic  Model  Generation 


In  many  complex  situations,  the  operator  can  use  incoming  data  to 
predict  the  general  trend  of  future  events  by  constructing  a  model  of 
his  environment.  The  question  addressed  in  this  section  is  how  well  the 
operator  can  generate  predictive  models  if  the  data  are  related  to  the 
general  trend  only  probabilistically.  The  optimal  strategy  in  such 
cases  would  be  the  least-square  solution  used  in  regression  analysis. 

For  a  medical  diagnosis  task,  subjects  were  able  to  learn  both 
linear  and  quadratic  trends  to  a  certain  extent,  but  they  did  better  at 
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predicting  linear  trends  (Brehmer  and  Qvarstrem,  1  ^  7t>  > .  Net  surprisingly 
the  less  variability  in  the  incoming  data,  the  better  subjects  were  able 
to  predict  trends  (Brehmer,  Kuv  lenst  ierna ,  and  .lergren,  19  74).  However, 
they  were  very  poor  at  learning  quadratic  trends.  They  were  even  worse 
at  using  the  trends  they  learned  when  they  had  to  predict  the  correct 
solution.  Again,  quadratic  trends  were  even  more  difficult  to  predict 
than  were  linear  trends. 

The  reason  for  the  latter  results  are  related  to  some  findings  by 
Hammond  (1972).  There  are  two  stages  to  conceptual  development:  acquisi 
tion  of  the  concept,  and  cognitive  control  or  output.  For  example,  one 
might  understand  that  a  series  of  numbers  increases  quadrat ical ly  and 
still  do  a  poor  Job  in  generating  the  sequence.  Hammond  was  able  to 
demonstrate  that  cognitive  control  was  more  difficult  than  concept 
acquisition  for  a  number  of  task  situations.  Also,  Hammond,  Stewart, 
Brehmer,  and  Steinmann  (1975)  found  that  subjects  with  the  same  basic 
cognitive  model  were  in  conflict  because  of  their  lack  of  output  control. 

Brehmer  (1974)  gives  evidence  which  indicates  that  the  operator 
uses  a  linear  trend  model  as  a  first  hypothesis.  Only  after  negative 
results  does  the  operator  begin  to  entertain  non-linear  relationships 
to  predict  future  trends.  if  a  subject  is  trained  to  recognize  non¬ 
linear  trends,  he  is  able  to  switch  to  linear  situations  rapidly. 

However,  the  reverse  is  not  true.  (Hammond  et  al,  1975.)  This  suggests 
that  the  operator  normally  entertains  a  very  simplified  model  of  the 
world.  This  model  may  be  functional,  simply  b  cause  simple  linear  trends 
do  a  good  job  of  predicting  many  types  of  complex  trends,  if  there  is 
a  monotonic  relationship  among  the  predictor  and  criterion  variables 
(i .e. ,  X  and  Y) . 

There  are  a  number  of  biases  reported  for  probabilistic  learning 
situations  (Lee,  1971).  The  most  prevalent  is  gambler's  fallacy  which 
has  been  reported  previously.  Another  common  bias  is  referred  to  as 
"probability  matching."  This  bias  involves  events  (e j . . .e^ . . ,en] 
occurring  with  specified  probabilities  Ipj . . .p^. . .pn] .  In  the  typical 
experimental  paradigm,  subjects  must  guess  the  next  ej  for  each  trial 
for  a  large  number  of  Independent  trials  (they  are  usually  told  the 
correct  ej  after  each  trial).  The  optimal  strategy  is  to  guess  the 
single  event  with  the  largest  probability  for  every  trial.  However, 
subjects  usually  guess  every  ej  with  the  same  approximate  relative 
frequency  as  its  relative  probability  (i.e.,  match  frequency  of  guessing 

*1  to  pj). 


Most  of  the  research  reviewed  indicates  that  the  ability  of  the 
operator  to  predict  future  events  based  on  uncertain  cues  is  poor. 
However,  Hammond  et  a  l  (1975)  reviewed  research  in  which  the  subjects 
were  quite  good  at  making  these  types  of  predictions.  This  suggests 
that  more  research  is  necessary  to  find  what  task  parameters  are 
important  determinants  of  the  operator's  ability  to  predict  future  trends 
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REAL-WORLD  TASKS 

The  purpose  of  the  previous  three  sections  was  to  characterize  the 
operator  in  complex  tasks  which  have  some  real-world  validity.  This 
section  will  report  operator  performance  characteristics  observed  in 
actual  real-world  (non- laboratory )  tasks. 


Scoring  Rules  and  Cal ib  rat  ion 

An  important  prerequisite  for  surveying  real-world  performance  is 
to  have  some  quantitative  criteria  for  judging  estimators.  Two  general 
approaches  have  been  used  to  assess  probability  estimators. 

1.  Proper  scoring  rules  are  computational  procedures  which  yield 
the  maximum  value  to  estimates  that  match  observed  frequency  distribu¬ 
tions.  Linear  payoff  schemes  (reward  linearly  related  to  number  of 
correct  outcomes)  are  not  proper  scoring  rules.  In  this  case,  the 
operator  can  maximize  his  score  by  assigning  1.0  to  the  most  probable 
category  (Murphy  and  Winkler,  1971).  Scoring  rules  using  logarithmic, 
quadratic,  and  spherical  functions  have  been  shown  to  be  proper  (Murphy 
and  Winkler,  1970).  Although  all  these  rules  yield  maximal  values  when 
predicted  frequency  matches  observed  frequency,  some  proper  rules  were 
more  sensitive  to  incorrect  estimates  than  others  (Murphy  and  Winkler, 
1971;  Brown  and  Shuford,  1973).  in  particular,  Hoffman  and  Peterson 
(1972)  found  a  quadratic  rule  to  have  only  minimal  impact  as  feedback, 
presumably  because  it  was  insensitive  to  estimates  that  were  not 
radically  different  from  the  correct  estimate. 

2,  Another  way  to  measure  the  operator’s  calibration  (i.e.,  how 
close  his  estimates  are  to  observed  estimates)  is  to  plot  observed 
frequency  of  events  against  the  operator’s  estimates  (Lichtenstein, 
Fischhoff,  and  Phillips,  1977).  This  has  the  advantage  of  showing  the 
direction  as  well  as  magnitude  of  operator  errors  (cf..  Figure  2).  As 
a  convention,  in  this  report  calibration  will  refer  to  plotted  data  and 
scoring  rules  will  be  understood  to  refer  to  any  of  the  proper  scoring 
rules. 

Operator  Performance 

Winkler  and  Murphy  (1973)  point  out  that  probability  estimation 
skill  in  the  real  world  involves  (i)  substantive  understanding  of  the 
events  being  predicted,  and  (2)  normative  understanding  of  the  under¬ 
lying  probability  structure.  In  other  words,  the  estimator  must  know 
the  dynamics  of  the  real-world  situation  before  he  can  relate  them  to 
a  probability  structure. 
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Weather  forecasters  must  combine  information  of  frequency  trends 
with  detailed  knowledge  of  meteorological  phenomena  in  order  to  predict 
probabilities.  in  general,  they  seem  to  do  a  good  job  (Lichtenstein, 
Fischhoff,  and  Phillips,  1977;  Winkler  and  Murphy,  1973).  In  one 
experiment,  their  scoring  rule  values  were  often  better  than  that  of 
a  Bayesian  solution,  because  they  understood  some  of  the  dependencies 
among  the  data  whereas  the  Bayesian  solution  wrongly  assumed  conditional 
independence  (Equation  3).  Some  forecasters  seemed  to  "hedge"  their 
results.  Hedging  involves  using  subjective  scoring  criteria  (improper 
ones)  which  weight  some  mistakes  as  more  disastrous  than  others  (e.g., 
a  farmer  who  does  not  prepare  for  frost  is  worse  off  than  one  who 
prepared  for  frost  which  did  not  happen)  (Murphy  and  Winkler,  1971). 

Forecasters  were  both  accurate  and  consistent  when  they  constructed 
credible  intervals  of  maximum  and  minimum  temperatures  (i.e.,  the  maximum 
would  fall  between  xi  and  X2  50%  of  the  time).  However,  when  they  were 
told  the  interval  and  told  to  assess  the  probability,  their  estimates 
were  excessive.  In  another  experiment,  they  assessed  area  probabilities 
(e.g.,  probability  of  rain  in  San  Francisco)  and  point  probabilities 
(e.g.,  probability  it  will  rain  at  station  M  within  San  Francisco). 

Their  results  indicated  some  statistical  oddities.  In  some  cases,  point 
probabilities  were  larger  than  area  probabilities  [P(A  u  B)  <  P(B)].  Also, 
some  forecasters  use  different  definitions  of  events,  making  it  difficult 
to  interpret  probability  estimates  because  their  meanings  are  variable 
(Murphy  and  Winkler,  1975). 

PEATMOS  is  a  computer  aid  which  aggregates  information  and  makes 
probability  estimates  for  weather  forecasting.  Murphy  and  Winkler 
(1975)  found  that  forecasters  were  not  influenced  by  knowing  the 
prediction  made  by  this  computer  aid.  Neither  PEATMOS  nor  the  fore¬ 
casters  showed  a  clear  edge  in  estimation  performance. 

Forecasting  is  an  ideal  estimation  situation  since  forecasters  are 
not  under  great  time  pressure  and  they  are  professionals  whose  daily  job 
involves  probability  estimation.  Thus,  they  are  not  as  likely  as  other 
estimators  to  resort  to  heuristics.  However,  even  their  performance 
shows  some  evidence  of  statistical  biases. 

Lichtenstein,  Fischhoff,  and  Phillips  (1977)  surveyed  human  perform¬ 
ance  in  a  variety  of  different  tasks  and  realistic  situations,  and  con¬ 
cluded  that  the  human  operator  was  poorly  calibrated.  The  operator  was 
more  likely  to  be  excessive  than  conservative,  but  he  showed  both  types 
of  errors. 

The  human  operator  performs  relatively  well  at  tasks  that  involve 
"configural  judgments"  (Anderson,  1972).  The  operator  is  apparently 
able  to  see  a  pattern  underlying  a  profusion  of  interrelated  cues 
Medical  diagnosis  involves  these  types  of  judgments.  However,  generally 
computers  are  no  worse  than  doctors  at  diagnosing  diseases  (Beach,  1975). 
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Stael  von  Holstein  (1972)  examined  stock  market  predictions  by 
stockbrokers,  bunkers,  statisticians,  etc.,  and  lound  them  to  be 
overconfident  (excessive)  in  their  estimation.  In  fact,  only  J  of  the 
72  subjects  did  better  than  would  be  expected  by  predictions  based  solely 
on  the  past  frequency  counts  of  the  events  being  assessed.  However,  the 
consensus  opinion  (l.e.,  the  weighted  average  of  the  best  subject’s 
predictions)  did  do  better  than  predictions  based  on  past  frequency  counts. 

In  a  simulated  military  command  situation,  computer-aggregated 
l’(dj)llk)  values  were  superior  to  operator  aggregation,  here  again,  the 
thoroughly  trained  operator  did  relatively  well  compared  to  the  computer. 

It  was  only  When  the  incoming  data  were  degraded  that  computer  aggrega¬ 
tion  was  clearly  superior  to  the  human  operator  (Sclium,  Goldstein,  and 
Southard,  1966)  . 

in  summary,  in  some  complex  situations  where  time  constraints  art' 
minimal  and  the  operator  has  substantial  knowledge  of  the  area,  his 
estimations  can  be  characterized  as  fair  to  good  (Beach,  197S;  Winkler 
and  Murphy,  1971).  There  also  are  a  number  of  realistic  tasks  and 
concomitant  situations  (time  constraints,  stress,  etc.)  in  which  the 
operator  can  be  character l zed  as  almost  a  "non-estimator,"  l.e.,  his 
estimates  not  only  are  poorly  calibrated,  but  also  do  not  follow 
normative  prescriptions  (Kahneman  and  Tversky,  1972a, b;  Lichtenstein, 
Flsehhof f ,  and  Slavic,  19  77). 

HELP  IN  ESTIMATING:  AIDING  AND  TRAINING 

Over  twenty  years  ago,  Simon  introduced  the  concept  of  bounded 
rationality.  Many  formal  decision  models  assume  a  rational 
dec  is  ion-maker  with  unlimited  processing  capabilities.  Simon  argued 
that  those  models  were  unrealistic,  since  constraints  on  human  compu¬ 
tational,  processing,  and  memory  abilities  made  human  decisions 
incompatible  with  normative  predictions.  However,  if  these  constraints 
were  understood  and  corrected  for,  then  normative  models  could  predict 
human  performance.  In  other  words,  the  human  operator  is  rational 
within  the  bounds  of  these  constraints. 

In  terms  of  the  biases  which  the  operator  evinces  in  probability 
estimations,  there  are  two  possible  ways  to  improve  operator  perform¬ 
ance:  (l)  the  operator  can  be  trained,  and  (2)  the  operator  can  be 

a  ided . 

Schaffer  (1976)  assessed  the  estimated  probability  distributions 
of  subjects  who  had  Just  finished  a  course  in  elementary  statistics 
at  the  university  level.  The  subjects  were  given  feedback  after  each 
session  and  presumably  understood  the  technical  issues  involved. 

However,  training  only  improved  their  probability  distribution 
estimations  to  a  marginal  extent.  Also,  the  subjects'  likelihood 
estimates  (for  Equation  2)  became  worse  during  training  and  their 
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posterior  odds  estimates  remained  at  the  same,  rather  poor  level  of 
performance.  It  seems  that  feedback  which  informed  the  subject  Lhat 
his  distributions  were  excessive  caused  him  to  be  too  conservative 
in  his  likelihood  estimates.  The  effect  of  feedback  seems  to  be 
complex,  making  the  estimation  training  of  operators  a  difficult  task. 

However,  Lichtenstein  and  Fischhot'f  (1978)  trained  subjects  using 
a  variety  of  feedback  indexes  including  scoring  rules,  hit  rate,  and 
personal  Interviews,  and  found  substantial  Improvement  in  their 
subjects'  calibration.  Since  almost  all  the  effect  took  place'  between 
the  first  and  second  session,  this  suggests  that  even  a  minimal  amount 
of  training  can  be  helpful  for  at  least  some  estimation  tasks. 

On  the  other  hand,  some  of  the  biases  seem  particularly  resistant 
to  training  (Fischhoff,  1975,  1977;  Lyon  and  Slavic,  197b;  Tversky  and 
Kahneman,  1977)  and  in  these  cases  aiding  the  operator  is  necessary. 

One  of  the  first  aids  developed  was  the  PIP  aid  mentioned  previously. 

This  aid  computes  posterior  odds  by  aggregating  likelihoods  estimated 
by  the  operator.  When  there  is  a  reasonable  amount  of  independence 
among  the  estimations,  this  aid  improves  performance  (beach,  1975; 

Domas  and  Peterson,  1972;  Schaffer,  1976).  Although  this  aid  extends 
the  computational  abilities  of  the  operator,  it  leaves  most  of  his 
biases  unc.orrected . 

Recently,  Slovic  (1978)  has  reviewed  a  number  of  aiding  techniques 
and  general  procedures  for  correcting  biases.  Kahneman  and  Tversky 
(1978a)  Introduced  specific  mathematical  functions  in  order  to  correct 
a  number  of  the  biases  they  discovered  in  their  earlier  papers.  Since 
these  functions  need  few  parametric  inputs,  they  couJd  be  computerized 
and  used  as  tactical  aids  (for  situations  where  immediate  decisions  are 
unnecessary).  More  complex  corrective  functions  have  been  suggested 
by  Llndley,  Tversky  and  brown  (l'*77).  These  functions  are  corrective 
procedures  for  probability  distributions  elicited  from  the  operator 
which  depart  from  probability  axioms  (e.g. ,  probability  of  all  possible 
events  do  not  sum  to  one) . 

The  three  studies  cited  above  represent  a  continuum  from  least  to 
most  sophisticated  aiding  procedures:  (1)  mostly  Insight  (Slovic,  1978), 
(2)  simple  corrective  functions  (Kahneman  and  Tversky,  1978a),  (3)  complex 
functions  (Llndley,  Tversky  and  brown,  1977).  The  usefulness  ol  these 
procedures  depends  on  both  the  particular  situation  and  the  training 
level  of  the  operator.  One  past  problem  with  decision  aids  (s  that  oper¬ 
ators,  particularly  those  in  leadership  positions,  are  not  apt  to  use  them 
if  they  do  not  understand  the  output  from  the  aid  (Slnaiko,  1977).  Thus 
simply  understanding  the  biases  (i.e.,  insight)  Is  the  necessary  t i rst 
step.  Next,  specific  corrective  procedures  can  be  explained  to  the 
operator.  If  the  operator  understands  the  necessity  and  the  mechanics 
of  these  procedures,  they  are  more  apt  to  use  a  preprogrammed  aid  which 
does  the  actual  computation.  Complex  procedures  such  as  those  suggested 
by  Llndley  et  al  (1977)  are  probably  more  useful  in  situations  where 
professional  decision  analysts  are  involved  in  the  decision-making  chain. 
Aiding  and  training  are  complementary  processes,  since  the  usefulness  of 
an  aid  Is  some  function  of  how  well  it  relutes  to  the  training  level  of 
the  operator. 
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SUMMARY  ANI>  0UT1.INK 


Tabic  1  1h  an  attempt  to  outline  the  01)0  In  terms  of  various  types 
of  decision  tasks  operators  perform  when  making  probability  estimates. 
The  first  two  columns  describe  the  characteristics  of  the  paradigms 
used  to  study  these  tasks.  Column  ;l  states  the  underlying  model  the 
operator  should  use  for  optimal  solutions.  Columns  4-7  characterize 
the  operator's  performance  for  these  tasks.  The  most  important  sources 
for  this  character  lest  Ion  are  given  in  Column  8. 

"Event  probability  estimation  f inexperienced)"  refers  to  the 
operator's  abilitv  to  estimate  either  probability  distributions  or  odds 
that  particular  events  will  occur.  It  differs  t rom  the  updating  task 
because  the  operator  may  have  little  knowledge  of  either  the  probability 
structure  ol  the  events  or  the  data  which  signal  the  events'  occurrence. 
This  task  more  closolv  approximates  operator  performance  in  situations 
in  which  he  simplv  does  not  know  the  important  parameters  of  the  situa¬ 
tion  and  vet  must  estimate  the  probability  of  an  event  occurring.  For 
example,  he  might  have  to  estimate  the  probability  that  route  A  is 
safer  than  route  H  the  first  time  he  flies  lit  a  hostile  zone  when  there 
has  been  little  advance  intelligence. 

On  the  othet  hand,  "event  probability  estimation  (expert)"  refers 
to  situations  where  the  operator  understands  both  the  underlying  dynamic 
ot  the  situation  and  is  somewhat  familiar  with  the  probability  structure 
The  operator  is  assumed  to  be  an  expert  at  combining  evidence  with  past 
frequency  counts.  The  time  stress  is  minimal.  It  differs  from  Lite 
updating  task  because  the  scenario  is  more  realistic  and  the  operator 
is  well  trained,  thus  understand  lug  the  possible  dependencies  which  can 
exist  among  real  world  data.  "Event  probability  estimations  (expert)" 
are  more  likely  to  be  made  during  mission  planning  than  in  mission 
execution.  Obviously  the  two  types  of  event  probability  estimations 
represent  two  extremes  on  a  continuum  rather  than  two  discrete  tasks. 
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WORTH  ASSESSMENT 


OVERVIEW 

Besides  assessing  Che  probability  of  an  event's  occurrence,  the 
operator  must  assess  the  worth  of  particular  outcomes.  This  assessment 
should  be  independent  of  probability  estimation.  The  first  step  in  the 
assessment  process  is  to  generate  a  list  of  possible  outcomes.  Outcomes 
[<(>1 . .  .<t>j . .  ,4>n)  are  results  of  actions  and  are  characterized  by  an  n 
dimensional  vector  [»^. .  .ai^*  •  »alnJ  with  each  dimension  representing 
a  facet  of  the  outcome's  worth  (Figure  5). 


ACTIONS  POSSIBLE  RESULTS  OUTCOMES 
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FIGURE  5.  Structure  of  the  Worth 
Assessment  Task. 


Formally,  worth  assessment  Is  the  evaluation  of  each  outcome  based 
on  one  or  more  dimensions  of  worth.  The  worth  assessment  process 
involves  different  scaling  techniques  which  imply  different  measurement 
models.  Worth  assessment  might  be  a  complex,  time-consuming  process  or 
a  "snap  Judgment."  The  purpose  of  this  section  is  to  examine  the  formal 
proparties  of  the  process  and  to  characterize  operator  performance. 

The  underlying  assumption  for  all  worth  assessment  is  that  the 
operator  has  a  particular  preference  structure  and  a  value  for  each 
outcome  can  be  generated  to  represent  it.  Generating  a  value  is  far 
from  trivial. 

The  pilot,  in  our  example,  must  assess  the  consequences  associated 
with  sinking  (or  not  sinking)  the  ship.  Obviously,  it  is  not  an  easy 
task.  He  must  know  what  dimensions  to  consider  and  the  ramifications 
of  each  outcome  for  these  dimensions.  These  ramifications  may  have 
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complex  second-  or  third-order  reverberations  (Slovic,  1979),  so  the 
pilot's  evaluations  may  tend  to  be  vague.  Table  2  represents  an  example 
of  possible  dimensions  and  their  evaluations  for  each  outcome.  However, 
as  presently  scaled  these  assessments  are  too  ill-defined  to  base  deci¬ 
sions  on.  Rather,  they  raise  a  number  of  questions.  First  of  all,  how 
valid  are  these  values?  What  is  moderate  compared  to  high  danger?  Is 
there  a  trade-off  between  danger  and  military  advantage? 


TABLE  2.  Worth  Values  for  Sinking  or  Not  Sinking 
Ship  Imaged  on  Display. 


Outcomes 


Worth  dimensions 

Enemy 

ship 

Friendly  ship 

Sink 

Not  sink 

Sink 

Not  sink 

A. 

Danger  level  to  pilot 

High 

Moderate 

None 

None 

B. 

General  military  situation 

Improved 

Same 

Worsen 

None 

C. 

Damage  cost  to  ship 

$1M 

0 

-$2M 

0 

D. 

Change  in  fuel  level 

-30  gal 

0 

-30  gal  1 

.  . 

0 

The  formal  procedures  designed  to  answer  these  and  related  issues 
are  reviewed  in  this  section.  Generally,  these  procedures  consist  of 
taking  both  the  verbal  and  numerical  values  Ln  Table  2  and  converting 
them  to  a  common  numerical  scale  which  reflects  their  relative  worth. 

The  actual  value  assigned  to  an  outcome  then  depends  on  the  measurement 
model  the  worth  assessor  feels  is  appropriate  to  the  situation.  In 
our  particular  situation,  this  process  may  seem  too  time-consuming  to • 
be  of  much  value. 

However,  even  in  a  time-constrained  environment,  the  operator  may 
informally  consider  different  dimensions  of  the  worth  assessment 
situation  and  mentally  make  trade-offs  among  dimensions.  Also,  with 
the  increasingly  sophisticated  software  capabilities,  in  attack  aircraft, 
the  actual  worth  functions  could  be  programmed  before  the  mission.  The 
pilot  could  then  enter  values  such  as  those  in  Table  2  assessing  the 
immediate  tactical  situation  and  the  computer  could  tabulate  outcome 
values  using  the  predetermined  functions  the  pilot  has  generated  before 
the  mission.  How  valid  such  a  procedure  would  be  depends  on  issues 
discussed  in  the  next  section. 

The  next  section  will  be  devoted  to  techniques  for  generating  worth 
functions.  Then,  measurement  models  these  techniques  are  based  on  will 
be  discussed.  Finally,  the  operator's  worth  assessment  characteristics 
will  be  examined. 
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SCALING  TECHNIQUES 
Methods 

Scaling  Is  the  process  of  assigning  numbers  to  items  (Coombs,  Dawes, 
and  Tversky,  1971).  Kneppreth  et  al  (1974)  have  prepared  an  extensive, 
nontechnical  guide  to  scaling  of  worth.  A  summary  of  their  data  is 
shown  in  Table  3.  Techniques  for  scaling  are  listed  below  to  give  some 
of  the  flavor  of  what  the  worth  assessment  process  consists  of. 

Ranking  Method?  In  this  method  ranks  are  assigned  to  outcomes  or 
an  indication  is  made  as  to  which  of  two  outcomes  is  preferred.  This 
gives  ordinal  scale  information. 

Equivalence  Method:  Outcomes  are  rated  as  fixed  categories  such 
as  good,  very  good,  etc.  This  technique  also  gives  only  ordinal 
Information. 


TABLE  3.  Comparison  of  Worth  Assessment  Methods.13 


Charactarl»tlc» 

Oirect 
me  t  hod  s 

Gamble 

methods 

Probabi  1 1 1  let 
requ ( red 

No 

No 

No 

Yes 

No 

Response  required 

Pretri ence 
iudgmen  t s 

Indifference 

i  id"'iit'iil  s  ! 

Qua  1  i  tail  i  ve 

•  Utl«l  vi.  1 

Indifference  judgments 
usually,  but  may  be  a 
preference  judgment 

Indif ference 

s  ad,U>>v  Ot  v 

Number  of  factor 
)ev«U  considered 

One 

two/ T hr ee 

Three /four  ^ 

Two 

Continuous  or  4is- 
crata  factor. 

01  sc  ret c  only 

Discrete  only 

l i t  her 

E  i  t  her 

Cont inuous 

Output 

Relat  ivr  worth 

Approximate  nu- 
merlcal  worth 

Numerical  worth 

Nume  r i ca 1  wor  t  h 

Rank i ng 

Case  of  use 

Very  simple 

Very  simple 

Maid  to  do  tor 
some  prop  1  e 

Tedious  ^ 

Tedious 

Training  of  dec  1  - 
9 ion  maker 

Almost  none 

Almost  none 

Moderate 

Extent i ve 

Little 

React  ion  of  dec i - 
%  ion  maker % 

Nigh  accept* 
ab i I  i  t  y 

High  accept  - 
abi li  t  y 

Often  requires 
sel 1 inq  for 

Accept Abi 1 i t  v 

Sometimes  trustrat inq 

Can  be  slow  end 
taxing 

Spaad 

Very  quick 

Very  quick 

Moderate 

Slow 

Slow 

Data  proctstlng 

Vary  little 

Vary  little 

Usua 1 1 y  none 

Moderate 

Can  be  extensive 

Unique  worth 
as* Ignment 

No 

No 

Yes 

Va> 

Yes 

Accuracy 

(Consistency) 

High,  if  only  a 
few  al ternat Ives 

High,  it  only  a 
few  al ternat ivex 

Moderate 

Accuracy  is  a  function 
of  patience,  ranqe  of 
indifference,  and  under¬ 
standing  of  probabili¬ 
ties  and  risk  properties 

Possible  wide 
ind If ference 

area 

Ski  1 1  and  axpar- 
lanca  raqulrad 
of  analyst 

Little 

Little 

Moderate 

High 

High 

4  From  Kn«ppr«th  •«  al . ,  197^. 
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Direct  Method:  The  levels  of  the  dimensions  are  assigned  numbers 
or  graphed  by  the  assessor  against  worth  values.  These  techniques  give 
interval  information. 

Gamble  Method:  Probabilities  are  varied  between  two  outcomes  until 
the  assessor  is  indifferent.  Alternately,  worth  levels  are  varied 
between  two  outcomes  differing  in  probability  until  the  assessor  is 
indifferent.  This  technique  gives  interval  information. 

Indifference  Methods:  Indifference  functions  are  graphed  or 
elicited  between  levels  of  two  dimensions.  litis  technique  can  give 
interval  information, 

Keeney  and  Raiffa  (1974)  and  Raifla  (1968)  discuss  gambling  and 
indifference  methods  which  are  more  sophisticated  than  the  methods 
outlined  by  Kneppreth  et  al  (1974) .  The  main  advantage  to  the  more 
sophisticated  methods  is  they  can  be  used  to  generate  worth  functions 
of  vectors  with  n  dimensions  rather  than  the  few  factors  considered  by 
Kneppreth  et  ai  (1974),  However,  these  methods  are  based  on  the  same 
basic  elicitation  procedures  as  the  reviewed  methods. 

MacCrimmon  (1968)  reviewed  a  number  of  methods  that  consider  the 
values  in  a  multidimensional  vector  without  exhaustively  considering 
all  dimensions  for  each  outcome.  These  arc  listed  below. 

Lexicographic :  Dimensions  are  ranked  in  importance.  A  subset  of 
outcomes  with  high  values  on  the  most  important  dimension  is  chosen. 

This  suhset  is  further  reduced  by  examining  the  next  most  important 
dimension  and  so  on  until  the  desired  outcome  is  chosen. 

Sat  1st  ic ing :  A  subset  of  dimensions  is  chosen  and,  if  these  values 
lead  to  a  clear  preference  structure,  the  outcomes  are  ordered.  However, 
if  evaluation  differences  among  outcomes  do  not  surpass  a  criteria  1 
level,  additional  worth  dimensions  are  examined. 

Maximln  and  Minimax:  Here  the  full  set  of  dimensions  are  considered, 
but  only  the  worst  level  among  the  dimensions  is  used  to  rank  the  desira¬ 
bility  of  the  ovi t c ome s . 

The  relevance  of  these  scaling  techniques  to  operator  performance 
depends  on  (1)  practicality,  (2)  formal  validity,  and  (3)  psychological 
relevance. 

Practicality  and  Scaling  Techn lques 

One  of  the  first  practical  problems  facing  life  operator  is  whether 
to  decompose  an  outcome  into  a  multidimensional  worth  vector.  In  many 
cases.  It  may  be  too  time-consuming  and  complicated  to  scale  each 
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dimension  rather  than  simply  assigning  a  value  to  each  outcome.  However, 
in  other  cases,  breaking  down  outcomes  into  their  important  dimensions 
helps  the  operator  structure  the  dynamics  of  the  problem  in  his  own  mind. 

A  practical  solution  to  this  problem  is  demonstrated  with  a  worth 
assessment  algorithm  developed  for  group  decision-making  (Leal,  Levin, 
Aamon,  and  Weltman,  1978;  Leal  and  Pearl,  1977).  The  algorithm  allows 
direct  assessment  of  outcomes  or  decomposition,  depending  on  the  situation. 

Outcomes  are  directly  evaluated  by  each  judge.  A  computer  sub¬ 
routine  examines  the  range  of  the  values,  performing  a  sensitivity 
analysis.  If  the  different  judges*  values  lead  to  the  same  decision, 
no  further  analysis  is  done.  On  the  other  hand,  if  the  divergence  of 
opinion  would  lead  to  different  decisions,  each  outcome  is  decomposed 
into  a  multidimensional  worth  vector.  The  values  of  the  dimension  are 
rated  by  each  Judge  and  discussed.  This  decomposition  allows  the 
decision-makers  to  identify  and  possibly  mollify  areas  of  discord.  The 
amount  of  scaling  done  by  the  decision-maker  is  not  pre-set,  but  rather 
it  is  the  minimum  necessary  to  arrive  at  a  consensus.  Sensitivity 
analysis  is  also  possible  for  individual  worth  assessments,  if  the 
assessor  gives  ranges  of  worth  and  not  a  single  value  lor  each  outcome. 

MacCrimmon  (1968)  suggests  combining  a  number  of  scaling  techniques 
in  order  to  simplify  complex  assessment  problems.  For  example,  a  com¬ 
bination  of  lexicographic  ordering  and  a  minimax  criterion  could  be  used 
to  eliminate  obviously  poor  choices.  The  A-7E  pilot  might  wish  to  consider 
2U  potential  targets.  He  can  choose  the  most  important  dimension  (e.g,, 
his  safety)  and  attempt  to  minimize  the  maximum  loss  by  eliminating  any 
targets  that  have  a  good  chance  of  shooting  him  down.  For  the  remaining 
subset  of  targets,  lie  can  consider  other  important  dimensions  before 
making  decisions.  The  important  point  is  that  these  worth  assessment 
techniques  are  not  mutually  exclusive:  the  best  technique  for  a  partic¬ 
ular  situation  depends  on  the  parameters  of  the  situation  and  the  ingenuity 
of  the  assessor. 

Validity  of  Scaling  Techniques 

The  validity  of  the  scaling  technique  depends  on  (1)  how  the  numbers 
were  generated  and  (2)  the  reasonableness  of  the  underlying  measurement 
model.  Considering  the  two-ship  problem  again  should  help  in  understanding 
these  two  concepts.  It  will  be  assumed  that  dimensions  are  additive. 

The  assessment  would  consist  of  decomposing  outcomes  into  a  worth  matrix 
(such  as  Table  2)  and  using  one  of  the  scaling  methods  to  assign  weights 
to  various  dimensions  and  numbers  to  various  levels.  Equation  9  could 
then  be  used  to  compute  the  worth  of  the  Kth  outcome  (e.g.,  sinking 
ship  B) . 
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"k  ■  E  bi  Dn  ^ 

i-l 

where 

Wg  *  value  of  the  Kth  outcome 

=  weighting  constant  for  dimension  i 

Djj  *■  value  for  the  jth  level  of  dimension  i  pertinent  to  the  Kth 
outcome. 

The  validity  of  the  worth  assessment  process  depends  both  on  the 
numbers  generated  and  on  the  appropriateness  of  Equation  9.  The  first 
criterion  (number  generation)  is  a  function  of  how  well  the  numbers 
represent  the  operator's  value  system.  Ranking  both  the  dimensions 
and  their  weights  results  in  losing  interval  information  concerning  the 
relative  importance  of  different  dimensions.  Direct  scaling  techniques 
do  not  force  the  operator  to  make  trade-offs.  The  assumption  is  that 
the  operator  can  generate  a  number  which  represents  a  valid  approxima¬ 
tion  of  the  dimension's  relative  worth.  Indifference  and  gambling 
methods  are  more  sophisticated.  They  generate  interval  information  by 
forcing  the  operator  to  make  indifference  judgments  between  worth  vectors 
by  either  changing  values  of  dimensions  or  manipulating  their  respective 
probabilities.  Those  methods  more  nearly  capture  the  essence  of  an 
additive  model  which  is  a  trade-off  function  (Keeney  and  Raiffa,  197(->; 
Raiffa,  1968). 

However,  the  advantage  of  using  the  more  sophisticated  methods  may 
be  offset  by  their  difficult  and  time-consuming  nature.  The  choice  of 
which  technique  to  use  depends  on  operator  training  level,  time  con¬ 
straints,  and  desired  precision,  as  well  as  on  the  theoretical  validity 
of  the  technique  (Kneppreth  et  al,  1974). 

The  second  criterion  involves  Equation  9,  which  is  based  on  an 
additive  conjoint  measurement  model.  The  properties  of  the  model  must 
be  realistic  in  terms  of  the  empirical  worth  assessment  situation. 

If  the  tvw-ship  problem  involved  correlated  dimensions  or  important 
interactions,  an  additive  model  could  distort  the  true  preference 
structure  of  the  operator.  The  axiomatic  basis  of  measurement  models 
and  their  behavioral  consequences  are  discussed  in  the  next  section. 


Psychological  Relevance 


The  ability  of  the  operator  to  use  these  scaling  methods  and  the 
processing  load  they  impose  on  the  operator  determine  their  psychological 
relevance. 
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Simply  ranking  dimensions  or  outcomes  involves  less  cognitive 
loading  than  does  weighting  dimensions  or  making  paired  comparisons. 

Such  a  simple  strategy  is  justified  for  many  decision  problems,  since 
empirical  and  analytical  evidence  indicates  that  more  complicated 
methods  often  result  in  the  same  decisions  (Eckenrode,  1965;  McClelland, 
1978). 


There  are  a  number  of  ways  to  generate  numbers  for  direct  scaling 
techniques  (Calanter,  1965;  Kneppreth  et  al ,  1974).  One  of  the  most 
controversial  is  the  ratio  method.  A  low  level  is  chosen  arbitrarily, 
and  other  levels  are  assigned  numbers  in  proportion  to  their  respective 
values  to  the  low  level.  Some  active  researchers  favor  using  this 
method  (Edwards,  1978),  whereas  Kneppreth  et  al  (1974)  argue  that  it 
is  psychologically  difficult  for  the  operator  to  assess  worth  in  this 
manner.  Therefore,  any  scaling  technique  considered  should  be  viewed 
with  some  caution.  A  technique  should  be  chosen  that  is  tailored  to 
the  individual  operator  by  being  simple  and  psychologically  unequivocal 
to  him. 

A  related  concern  is  whether  the  very  use  of  formal  scaling  models 
distorts  the  decision  situation.  It  is  possible  that  the  values  elicited 
during  such  a  procedure  are  unstable  and  represent  a  conciliatory  attempt 
by  the  operator  to  cooperate  in  the  scaling  process.  This  is  particularly 
possible  when  the  values  are  elicited  from  the  operator  by  a  decision 
analyist.  The  analyist's  efforts  to  simplify  the  decision  situation 
may  result  in  structuring  the  problem  in  a  manner  that  is  totally  un¬ 
representative  of  the  situation  as  seen  by  the  operator.  One  way  to 
circumvent  this  is  to  have  the  operator  generate  an  informal,  intuitive 
analysis  of  the  decision  situation  before  any  formal  analysis  is  done. 
Then,  if  the  operator's  informal  model  differs  from  the  formal  scaling 
solution,  the  operator  can  rethink  the  problem  and  attempt  to  reconcile 
these  discordant  assessments.  Using  this  strategy,  the  operator  is  not 
driven  by  the  rigid  formalism  of  some  of  the  scaling  techniques,  but  is 
able  to  use  his  own  intuitive  judgement  in  order  to  help  guide  the  worth 
assessment  process.  In  general,  any  scaling  procedure  that  passively 
elicits  values  from  the  operator,  runs  the  risk  of  dictating  rather 
than  discovering  an  underlying  preference  structure.  (Fisehoff,  1978; 
Fischoff,  Slovic  and  Lichtenstein,  in  press.)  Fisehoff  (in  press) 
has  recently  compared  worth  assessment  methods  to  clinical  methods  in 
psychology.  The  analogy  is  apt  since,  when  both  methods  use  insight  and 
flexibility,  the  results  are  usually  very  useful.  However,  when  the 
analyist  is  wedded  to  formal  doctrine  and  inflexible  models  in  either 
discipline,  the  resulting  analysis  often  does  more  harm  than  good. 

One  purpose  of  decomposing  outcomes  into  worth  vectors  is  to  reduce 
the  operator's  cognitive  load.  Assessing  outcomes  involves  mentally 
integrating  information  over  a  number  of  dimensions  and  levels.  Decom¬ 
position  allows  the  operator  to  assess  worth  in  smaller,  more  manageable 
chunks  (Fischer,  1975;  Miller,  1956). 
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MEASUREMENT  MODELS 
Importance 

Measurement  procedures  (such  as  Equation  9)  imply  that  the  numbers 
being  used  have  certain  properties.  The  implicit  properties  underlying 
the  measurement  procedures  are  its  axioms  (Krantz,  Luce,  Suppes,  and 
Tversky,  1971).  A  measurement  model,  then,  is  a  collection  of  axioms 
that  allow  specified  numeric  procedures  to  take  place  among  events 
defined  by  the  model.  Models  are  useful  because  they  allow  manipulation 
and  abstraction  of  real-world  processes.  Pearl  (1977)  points  out  that 
worth  assessment  is  related  to  specific  neurological  activities  which 
govern  the  operator’s  preference  structure.  Measurement  models  are 
the  framework  in  which  these  activities  are  assessed  and  assigned  numbers. 

The  decision-making  characteristics  of  the  operator  depend  on  how 
well  the  particular  worth  assessment  model  he  uses  reflects  his  preference 
structure.  Part  of  the  operator's  task  is  c i rcumscribed  by  the  model's 
axioms.  If  his  performance  is  incompatible  with  some  of  the  axioms, 
then  particular  measurement  procedures  may  be  inappropriate. 

Coombs,  Dawes,  and  Tversky  (1971)  make  the  distinction  between 
"simple"  and  "existential"  axioms.  The  former  deal  with  the  model’s 
specific  assumptions,  whereas  the  later  involve  conditions  such  as 
"continuity"  dealing  with  properties  of  number  systems  in  general.  The 
present  discussion  will  focus  on  "simple"  axioms  which  have  important 
behaviorial  consequences;  all  necessary  axioms  for  a  particular  model 
will  not  be  discussed. 

A  thorough  discussion  of  measurement  models  is  beyond  the  scope  of 
this  paper.  The  focus  of  this  section  will  be  on  (1)  how  realistic  the 
assumptions  of  the  model  are  in  relation  to  operator  performance  and 
(2)  the  usefulness  of  procedures  based  on  the  model  for  the  worth 
assessment  tasks.  For  a  complete  discussion  of  the  mathematical  and 
logical  foundations  of  these  models,  the  reader  is  referred  to  Coombs, 
Dawes,  and  Tversky,  1971  and  Krantz,  Luce,  Suppes,  and  Tversky,  1971. 

Ordinal  Preference  Models 

Ordinal  preference  models  are  the  basis  of  scaling  techniques  that 
measure  the  operator's  preference  structure  on  an  ordinal  scale. 

The  first  axiom,  A.l  (connectivity),  indicates  that  an  ordinal 
number  can  be  assigned  to  any  two  outcomes  (x  or  y)  indicating  preference 
or  indifference  (Luce  and  Suppes,  1965). 


x  >  y 


or  y  i  x 


(A.l) 
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This  class  of  models  also  allows  for  the  use  of  strict  Inequality 
operators  (>,<)  where  clear  preference  structures  are  evident.  A  second 
axiom,  A. 2,  requires  that  relations  among  outcomes  be  transitive. 
Transitivity  means  that  if  x  is  preferred  to  y  and  y  to  z  then  x  must 
be  preferred  to  z. 


x  >  y  and  y  >  i  then  x  >  z  (A. 2) 

Because  of  variability  in  human  performance,  A.l  and  A. 2  are  certain 
to  be  violated  in  many  real-life  tasks.  Tversky  (1969)  suggests  that  a 
weak  stochastic  transitivity  condition,  A. 3,  is  the  limiting  factor  for 
determining  whether  the  assessment  process  is  based  on  an  ordinal 
preference  model. 


lx,y)  >  1/2  and  [x,z]  >  1/2  then  {x,z]  >  1/2  (A. 3) 

where  (x,y]  means  x  is  preferred  to  y. 

Axiom  A. 3  indicates  that,  if  the  operator  examined  three  outcomes 
a  large  number  of  times,  his  orderings  would  be  transitive  more  than 
half  the  time.  Most  experimental  results  support  models  that  are 
based  on  at  least  weak  stochastic  transitivity  (Luce  and  Suppes,  1965; 
Tversky,  1969).  However,  for  experiments  with  a  large  number  of  possible 
outcomes,  only  about  25%,  at  the  most,  of  the  relationships  can  be 
intransitive.  Thus  these  tests  may  be  insensitive  to  some  patterns  of 
intransitivity,  because  the  patterns  would  show  up  in  the  data  analysis 
very  rarely.  Tversky  (1969)  examined  situations  where  he  suspected 
intransitive  relationships,  and  his  results  supported  the  hypothesis 
that  the  human  operator  is  intransitive  under  certain  conditions. 

Table  4  illustrates  a  condition  similar  to  those  in  which  Tversky 
found  the  operator  to  be  systematically  intransitive. 


TABLE  4.  Possible  Intransitive  Relations  Among 
Ship-Destruct ion  Preferences. 


Worth  dimensions 

Ship 

Enemy  sh i p 
cost,  $M 

Danger  level 
to  pilot 

A 

9 

Low 

B 

10 

Moderate 

C 

12 

High 
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The  operator  would  choose  to  sink  A  rather  than  B,  and  B  rather  than 
C,  but  C  rather  than  A.  The  rationale  of  the  operator  is  that  cost  is 
the  roost  important  dimension,  but  that,  unless  the  values  in  that  dimen¬ 
sion  are  at  least  $3M  apart,  the  preference  would  be  based  on  danger. 

In  such  a  case  no  rank  ordering  of  the  outcomes  is  possible;  thus  any 
of  the  scaling  techniques  based  on  ordinal  information  are  invalid. 

Such  an  example  is  not  contrived;  there  is  other  experimental 
evidence  to  support  the  notion  that  values  must  be  a  certain  critical 
distance  apart  before  clear  preferences  exist  (Coombs,  Dawes,  and  Tversky, 
1971;  Suppes  and  Walsh,  1959).  Also,  polling  records  show  clear  cases 
of  systematic  and  not  just  nonstochastic  intransitivity  (Lee,  1971). 

This  does  not  indicate  that  the  operator  is  necessarily  irrational, 
only  that  worth  assessment  may  involve  heuristic  rather  than  strictly 
analytical  processes.  The  operator  with  a  limited  processing  capacity 
may  not  be  able  to  examine  multidimensional  worth  vectors  for  each 
outcome.  Thus  he  may  lexicographically  search  dimension  by  dimension, 
eliminating  outcomes  with  less  than  optimal  values  on  important  dimen¬ 
sions.  Tversky  (1972  a,  b)  provides  data  to  support  such  a  model  of 
outcome  preference,  which  he  refers  to  as  "Elimination  by  Aspect." 

This  method  may  not  be  ideal,  since  important  information  in  the 
multidimensional  worth  vector  is  ignored;  however,  it  may  be  efficient 
considering  the  operator's  processing  limitations. 

Ordinal  preference  models  are  the  simplest  measurement  models  for 
worth  assessment.  However,  even  for  this  relatively  simple  case, 
operator  performance  is  not  always  commensurate  with  ordinal  models. 

In  certain  circumstances  (Luce  and  Suppes,  1965;  Tversky,  1969), 
heuristic  assessment  procedures  might  be  employed  by  the  operator  which 
violate  transitivity  assumptions.  The  main  reason  for  this  seems  to  be 
the  operator's  processing  limitations  rather  than  inherent  irrationality. 

Conjoint  Measurement  Models  (Additive  and  Multiplicative) 

Conjoint  models  are  the  basis  of  scaling  techniques  which  measure 
the  joint  effect  of  two  or  more  dimensions  on  the  worth  of  an  outcome. 
These  models  presuppose  an  ordinal  model,  but  involve  additional  assump¬ 
tions  resulting  in  interval  information  concerning  worth.  Decomposition 
of  an  outcome  into  different  additive  dimensions  is  based  on  an  additive 
conjoint  model.  The  following  two  definitions  represent  the  simplest 
additive  model  having  two  dimensions  x  and  y,  each  having  three  levels: 
a,  b,  c  and  d,  e,  f,  respectively  (Coombs,  Dawes,  and  Tversky,  1971). 
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<Had)  -  h(a)  +  q(d)  (D-l) 

<Kad)  i  0(be)  If  M(ad)  .i  M(be)  (D-2) 

where 

(xiyi)  "  vector  containing  the  1th  levels  of  x  and  y 
4>,h,q  -  function**  defined  on  x  and  y 
M(x1y1)  »  measured  worth  of  (x^y|) 


The  first  definition  states  that  a  vector  is  decomposable  Into  two 
additive  functions  representing  levels  of  Its  Important  dimensions.  The 
second  definition  indicates  that  the  decomposable  function  has  the  same 
ordinal  properties  as  the  measured  effect  of  the  nondecompoaed  worth 
vector. 


Table  5  Illustrates  these  two  definitions.  The  levels  (a,  b,  c 
and  a,  e,  f)  of  the  two  dimensions  (x  «*  ship  cost;  y  -  danger  to  pilot) 
are  assigned  numbers  reflecting  their  relative  worth.  These  values  are 
listed  as  marginal  scale  rates  in  the  table,  whereas  the  cell  values 
reflect  their  measured  Joint  el  feet.  These  definitions  are  satisfied 
If  (1)  the  cell  values  are  some  additive  function  of  the  scaled  values 
of  cost  (x)  and  danger  (y),  and  (2)  the  value  assigned  an  outcome  Is 
commensurate  with  tlu*  operator  rankings  of  these  outcomes.  These  del  tui¬ 
tions  Imply  two  types  ot  Independence. 


The  first  type  of  Indepeudenee  requires  that  the  marginal  scale 
values  uniquely  determine  the  measured  conjoint  effect.  This  means 
that  the  dimensions  do  not  have  an  interactive  el  feet  when  combined. 
The  second  type  of  independence  Implies  that  different  levels  of  the 
dimensions  are  separately  realizable.  This  refers  to  conditions  where 
having  a  particular  value  on  x  does  not  determine  the  level  of  y. 

For  example,  this  type  of  independenco  would  be  violated  if  a  value  of 
hLgh  danger  precluded  a  value  of  low  cost  as  a  possible  outcome. 
(Krantz,  l.uce.  Sup  pea,  and  Tversky,  1971) 
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TABl.K  S.  Additive  Conjoint  Kllects  of 
Levels  of  I)  linens i ons  x  and  v.* 


*  Scaled  values  are  some  weighted  tunot ions 
of  tlio  decomposed  valuos  generated  din  ing  (ho 
worth  assossmont  process  face  Koonv  and  Kaitta, 

19761  and  coll  onlrlos  aro  a  moasuro  of  the  joint 
of  foot  of  xj  and  yi  (t.e.,  Mlxt  y^M. 

Other  nooossary  properties  of  this  model  include  solvability  and 
cancellation.  Solvability  assures  that  ther*  are  sufficient  levels  ot 
the  dimensions  to  compute  a  solution.  Cancellation,  A. 4,  is  an  algebraic 
result  of  the  additive  combinations  of  certain  levels  and  can  be  inter¬ 
preted  as  a  special  case  of  transitivity. 

It  M(ae)  2  M(bd)  and 

M(bt")  2  M(ce)  then  lA.s) 

M(af)  2  M(cd) 

This  relat  ionship  is  not  obvious  and  results  from  the  fact  that,  it 
additivity  holds,  then  N(ao)  2  M(bd)  and  M(bt )  2  Mice)  can  be  decomposed 
into  additive  functions  and  rearranged  into  Kipiat tons  10  and  10*. 

hfa)  -  qfd)  2  wfb)  -  gfe)  110) 

w(l>)  -  gfe)  2  Ifc)  -  lit)  110') 
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This  roarrangumont  indio.it  os  tii.it  In  .  au-.o  ot  ii.msinvih  in*  '  ll  4uiu> 
(s  trui*: 


i>  f.l)  -  q  (it  )  _  l  (o  )  -  t  t  t  '  ill'll 

Koa  rrann  i  n>;  tliis  oqu.it  i«*n  inti*  it',  i.lditiv.*  i  ,m  m  tioh>  i-iq-.i.  .  « ■  t  *  t  ,  m 
dl'l  In  It  toils  1  fll-lt  .nivl  tl'-.M  .no  line,  tlio  ii'.uit  .‘I  V  \  i  op  .  iiO,  i 
tu'opssar l 1 v  lollows. 

Ii  (a )  +  r  ( t  )  _  It.)  +  q  td )  t  till*  t 

Mt.it  )  _  Mud)  ii''4  ' 

Til  o  I  ao  t  that  I  1 1  i  s  rolat  i  oii.sli  i  p  is  not  nil  hi  l  nr  nukrs  <•  \  pi  a  tmou  t  a  I 
tosts  all  t  ho  inoro  oruotal.  C>’i'ii*l’s  and  Komot  i t a  * s  i  I  ■•>.'* )  suh  toils 
ova  l  tut  toil  thi*  worth  ol  outoomos  vlitioi  inp  on  t  ho  diniousioii  ot  pi  ohal*  i  1  i  t  v 
and  monot  ar  v  pavoft  .  Thoir  suhjoot  's  ordoi  i  nr,  ot  outtomos  was  ,  onmiou - 
surato  with  t  ho  oamol  lat  ion  axiom  in  .‘9  ot  tl »  oasos.  Adams  and  I'apot 
(I9r>9)  found  that  most  ol  thoir  sub  loots  (SO"..)  ratod  jot*  a|q>li<auts  in 
ai'oot'ila  iKO  wi  (  Ii  tiio  oanooHut  ion  axiom  most  ot  tho  i  t  mo  (9f>  ).  Whoioas, 
I’isi'lior  *s  (I97t>)  rosults  woro  similar  tor  ratinp.s  ol  job  ojqioi  i  mi  i  I  i  os  , 

Thoso  studios  suppost  that  oamollat  ion  is  a  i  oasonuh  I  o  oondit  ion 
for  at  loast  t  ho  simp  to  paradip.ms  roviowod.  Hooauso  ot  sub|oot  vat  la¬ 
bility,  thoso  axioms  an*  only  an  approximato  dosoription  ol  opotatoi 
pol  l  ormaiioo.  Ilowovor,  t  hov  si  i  1  1  Ii.ivo  provod  to  ho  pood  ptodiolors  ol 
operators’  dooisioiis  in  a  numhor  ol  oxpoi  imontnl  p.trudipms  (I'isohot  . 

19/'*.  1 9  7 b  )  . 

There  aro  somo  oasos  whon  i  ho  nudot  I  v  i  n>;  i  o  1  a  t  i  onsh  i  ps  a  mo  up  worth 
dimi'iisions  aro  simplv  not  additivo.  Important  int  or.n  I  ions  (ospoi  iallv 
nonmono  t  lUiio  ones)  and  t  ho  situat  ion  whot  o  worth  booomos  -ot  o  it  one 
dimonsion  is  /or o  aro  oxamplos  ot  snoh  oasos  tlluhot  ,  I ** / •’» ;  Wintortoldt 
and  I'isohor,  197'*).  It  oan  ho  shown  that,  it  tho  valuos  ol  d  intons  ions 
aro  prof  oront  i  a  1  I  y  indopondont  fsi’o  Koonoy  and  Kaitta,  |9/<>),  a  multi- 
plioatlvi*  conjoint  modi*  I  oan  ho  usoil  to  ova  luat  o  wort  It  (Koouoy  and 
Kaifla,  19/h;  Koenev  and  Sioltornt.in,  19,’u).  Kin  1  hormoi  o,  oxpotimontal 
ov  Ivlonoo  suppost  s  tliat  add  i  t  ivo  modi*  Is  may  ho  too  simplo  to  modol  tho 
oporator's  proloronoo  M  mot  hi  o  in  many  worth  ovalu.it  ion  situat  ions 
(Andoi'son,  |9li‘>,  l'i/’();  Andot  son  and  Slianti'.iu,  19/7;  Ciassor  and 
Amlorson.  1 9  7 A ) . 
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•anqPA  uoqaaqqaa  aqq 

qqqrt  paqRqaaaoa  Xqq8qq  st’«  lapoui  aqq  uaqm  uaao  aoaaa  snoqaas  oq  sppaq 
sassaaoad  qpuoqqaunq  Xpnqs  oq  sjapotu  uoqssaaSaa  8uqsn  qpqq  papnqauoa  put? 
qaapasaa  ipaq8oioqoXsd  qo  spaap  qo  aaquinu  p  paXaAans  (^^6[)  npaqupqs  put? 
uosaapuy  *iapoui  apauqq  aqduiqs  p  qqq«  uoq  qpq  aaaoa  q8qq  p  Moqs  jqqqs 
qnq  ‘xaqduioa  Xqqeuoqqaunq  aq  Xtnn  aanqanaqs  aauaaaqaad  8uqXqaapun  aqx 
‘  (996T  *q5TUIS  pue  aaddpaQ)  sqapoui  qimoq qatinq  jaqqpj  aAq qaq pnad  aap  sqapoui 
uojssaaSafl  '(IZ6T  ‘uqaqsuaqqaqq  pup  aqAoqs  i^Z6T  ‘npaqupqs  pun  uosaapuy 
•<!Z6I  ‘uosaapuy)  sajnpaaojd  qpuoq qpqaaaoa  jo  asn  aqpq adoaddpuq 
lit?  sq  sqqq  jpqq  qtto  paqu j  od  aAPq  sjaqajpasaj  quaaaa  ‘aaAawoq  ‘  (TZ61 
* uqaqsuaqqaqq  put?  aqAoqj; )  sassaaoad  qpquauiSpnf  no  qaapasaa  Xqapa  ux 
saqaqqod  qqjOM  ,sa8pn("  aqq  aanqdpa  oq  pasn  aaaw  sqapoui  uoqssaa8a>q 

•pqt?p  aqq  uiojj  paAqaap 

sqqq  saaRnbs-qspaq  aap  aoqaaA  qqaow  aqq  uq  suoqsiiaiuip  aqq  aoq  sqqSqaw 
aqq  pup  joqwjado  aqq  Xq  paqpjauaS  sq  auioaqno  up  joj  anqPA  qqaow  aqq 
‘aspa  sqqq  ui  •  saauaaa jajd  qtmqap  s',joqt?jado  aqq  uo  paspq  sqapoui  qnaq  q 
-stqpqs  asn  oq  sx  ssaaoad  quauissassp  ipjon  aqq  qapoui  oq  Xt?«  aaqqouy 

s'j~poW  fpafqsxTpqs 

•  suoxsuauixp  owl  upqq  aaoui  qqjM  saoqaaA  aoq  xaxdutoa  ssaq  auioaaq  sxapoui 
quqofuoa  ‘sXpm  autos  uq  ‘snqx  •  suoq qt?uxq«ioa  aqqqssod  aqq  qo  qasqns 
X  |  puts  p  Xquo  tfuququipxa  Xq  quapuadapux  X q  q- '>TUU  oap  MaqqM  suoqsuaiuqp 
Suqqnaqas  jo  spoqqaui  aqpjqsuouiap  pqqqt?N  put?  Xauaa>q  ‘  suoq  qpuq  qiuoa 

uoqsuauiqp  aqqqssod  q-u-u<;  oat?  aaaqq  qSnoqqqp  ‘osqy  *  (XZ61  ‘XqqsaaAi  pup 
‘saddns  ‘aanq  ‘zquuaqq)  Xjpssaaau  aaSuoq  on  sq  uoqqpq ) aaupa  qo  Xqaadoad 
aqq  ‘o«q  upqq  aaqaaafl  sq  (u)  suoqsuamqp  qo  aaqtunu  aqq  uaqw  sasna  joj 
•suoqsuauiqp  oi*vq  qqq«  aoqaaA  qqao/n  p  pauinssp  spq  uoqRsnasqp  sqqx 

’(9£6I  ‘epat?«p:x  pup  aauqpap;)  !g£6q  ‘spauwpa)  uoqsqaap  amps 
aqq  oq  paaq  pqnort  sqapoui  qqoq  asnpaaq  Xqdtuqs  sassaaoad  xaqdutoa  aaoui  qo 
suoq qpuiqxoaddp  aqpnbapp  aq  oq  aAoad  sqapoui  aAqqqppp  ‘sarnqq  Xupqq  *qapoui 
aAqqpaqqdqqqnui  p  Suqsn  jo  XqjAqqaqpaad  pappp  aqq  pup  sqapoui  aAqqqppp  qo 
Xqqaqqduiqs  aqq  uaa«qaq  qqo-appaq  p  sq  aaaqq  *sqsXqpup  qt?uqq  aqq  uq 

•qapoui  aAqqpjqqdqqqnui 
r  8uqsn  Xq  a8upqa  Xqqt?qaaaddp  aoq  paqunoaap  aaupqaPA  qo  qunouip 
aqq  pqp  uoqqpnqqs  uqpqaaaun  aqq  uq  Xquo  ‘aaAawoH  • sqapoui  quqo fuoa 
aAqqqppp  Xq  upqq  aAqqpaqqdq qqnui  Xq  paqqaasap  aaqqaq  aaa«  suoqqpnqpAa 
qqaoft  , sqoafqns  sqq  qo  qqpq  qttoqR  saspa  qqoq  uq  •  squauiuoaqAua  uqpqjaa 
pup  uqpqaaaun  qqoq  uq  quauissassp  qqaow  paqpnqs  (9^6T)  -toqosq^ 
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Fischer  (1976)  studied  worth  assessment  in  both  uncertain  and 
certain  environments.  In  both  cases  about  half  of  his  subjects'  worth 
evaluations  were  better  described  by  multiplicative  than  by  additive 
conjoint  models.  However,  only  in  the  uncertain  situation  did  the 
amount  of  variance  accounted  for  appreciably  change  by  using  a 
multiplicative  model. 

In  the  final  analysis,  there  is  a  trade-off  between  the  simplicity 
of  additive  models  and  the  added  predictivity  of  using  a  multiplicative 
model.  Many  times,  additive  models  prove  to  be  adequate  approximations 
of  more  complex  processes  simply  because  both  models  would  lead  to  the 
same  decision  (Edwards,  1978;  Gardiner  and  Edwards,  1975)  . 

This  discussion  has  assumed  a  worth  vector  with  two  dimensions. 

For  cases  when  the  number  of  dimensions  (n)  is  greater  than  two,  the 
property  of  cancellation  is  no  longer  necessary  (Krantz,  Luce,  Suppes, 
and  Tversky,  1971)  .  Also,  although  there  are  2n-n-l  possible  dimension 
combinations,  Keeney  and  Raiffa  (1976)  demonstrate  methods  of  selecting 
dimensions  which  are  mutually  independent  by  examining  only  a  small 
subset  of  the  possible  combinations.  Thus,  in  some  ways,  conjoint 
models  become  less  complex  for  vectors  with  more  than  two  dimensions. 

Statistical  Models 


Another  way  to  model  the  worth  assessment  process  is  to  use  statis¬ 
tical  models  based  on  the  operator's  actual  preferences.  In  this  case, 
the  worth  value  for  an  outcome  is  generated  by  the  operator  and  the 
weights  for  the  dimensions  in  the  worth  vector  are  least-squares  fits 
derived  from  the  data. 

Regression  models  were  used  to  capture  the  judges'  worth  policies 
in  early  research  on  judgmental  processes  (Slovic  and  Lichtenstein, 

1971)  .  However,  recent  researchers  have  pointed  out  that  this  is  an 
inappropriate  use  of  correlational  procedures  (Anderson,  1972; 

Anderson  and  Shanteau,  1977;  Slovic  and  Lichtenstein,  1971).  Regression 
models  are  predictive  rather  functional  models  (Drapper  and  Smith,  1966). 
The  underlying  preference  structure  may  be  functionally  complex,  but 
still  show  a  high  correlation  with  a  simple  linear  model.  Anderson 
and  Shanteau  (1977)  surveyed  a  number  of  areas  of  psychological  research 
and  concluded  that  using  regression  models  to  study  functional  processes 
leads  to  serious  error  even  when  the  model  was  highly  correlated  with 
the  criterion  value. 


On  the  other  hand,  the  use  of  linear  regression  models  to  predict 
worth  judgments  has  proved  to  be  very  successful  (Dawes  and  Corrigan, 
1974;  Goldberg,  1970).  Bootstrapping  is  the  use  of  a  regression  model 
to  predict  decision  rules  based  on  operator  performance.  In  general, 
bootstrapping  models  perform  better  than  the  judges  they  model  (Dawes 
and  Corrigan,  1974)  because  the  variability  reflecting  lack  of  fit  due 
to  a  linear  model  is  often  less  than  the  variability  due  to  the  judges' 
inconsistencies  (Goldberg,  1970). 

Hamner  and  Carter  (1975)  used  four  regression  models  based  on  their 
subjects'  estimates  to  predict  ideal  production  rules.  These  models 
were  linear  combinations  of  different  information  sources  such  as  sales 
forecasts,  current  inventory,  etc.  All  four  models  led  to  production 
rules  better  than  those  chosen  by  the  subjects  whose  data  were  used  to 
generate  the  rules.  Thus  the  weights  inferred  from  the  subjects' 
performance  reflected  a  more  stable  indication  of  their  worth  vector 
than  did  the  decisions  made  by  the  subjects.  Dawes  and  Corrigan  (1974) 
reviewed  experiments  in  which  bootstrap  models  of  judges  were  more 
efficient  than  the  judges'  decisions  in  almost  all  cases. 

Perceptronics  (1977)  has  recently  developed  a  decision  aid  (ADDAM) 
which  incorporates  features  of  bootstrapping  models.  The  weights  of  the 
worth  vector  change  as  a  function  of  operator  decisions  and  are  updated 
using  an  online  computer  program.  Thus  the  operator  has  a  computer 
model  of  his  worth  vector  which  can  be  used  to  select  optimal  decisions. 
Preliminary  experimental  evidence  indicates  that  this  aid  improves 
operator  performance. 

Even  simpler  than  fitted  regression  models  are  models  with  unit 
weights  for  each  dimension.  Dawes  and  Corrigan  (1974)  give  evidence 
that  such  models  result  in  predictions  that  are  almost  as  good  as 
least-square  weights,  provided  there  are  no  significant  nonmonotonic 
interactions.  This  would  greatly  simplify  the  operator's  task,  since 
generating  an  additive  model  such  as  Equation  9  would  consist  of 
simply  normalizing  the  levels  of  each  dimension  and  then  summing. 

However,  McClelland  (1978)  provides  analytical  arguments  which 
show  that  unit  weight  models  are  most  likely  to  be  seriously  erroneous 
for  nondominated  outcomes.  Nondominoted  outcomes  are  those  for  which 
the  operator  must  determine  trade-off  functions  among  dimensions  in 
order  to  choose  the  outcome  with  the  largest  worth.  Since  nondominated 
outcomes  are  the  most  important  for  worth  assessment,  unit  models  would 
seem  to  have  limited  usefulness  except  in  situations  where  the  weights 
of  the  various  dimensions  ire  unknown. 

Regression  approaches  extend  the  usefulness  of  using  a  worth  vector 
by  allowing  weights  for  various  dimensions  to  be  determined  by  operator 
performance.  The  ultimate  usefulness  of  these  models  depends  on  how 
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well  they  predict  worth  assessment  in  novel  situations.  Unfortunately, 
most  of  the  models  were  tested  on  the  data  used  to  generate  the  model 
(Hamner  and  Carter,  1975). 

Other  Measurement  Models 


A  number  of  multidimensional  measurement  models  have  been  developed 
that  depend  on  the  concept  of  psychological  distance  (Coombs,  Dawes, 
and  Tversky,  1971).  The  worth  models  posit  ideal  points  in  multi¬ 
dimensional  psychological  space.  Different  metrics  are  used  to  measure 
the  distance  of  worth  vectors  to  these  ideal  points.  The  closer  the 
vector  to  the  ideal  point,  the  greater  the  worth.  Although  scales 
based  on  distance  models  have  been  used  in  many  areas  of  psychological 
research  (Coombs,  Dawes,  and  Tversky),  quite  a  few  of  those  scales 
violated  the  axioms  they  were  based  on  (Beals,  Krantz,  and  Tversky,  1968) . 

There  has  not  been  too  much  use  of  distance  models  for  worth  assess¬ 
ment,  probably  because  these  techniques  have  more  theoretical  than 
practical  appeal.  Klahr  (1969)  measured  different  dimensions  of  college 
admissions  standards  (e.g.,  college  boards,  I.Q.,  etc.)  using  the  concept 
of  psychological  n-d imensional  space  with  ideal  points.  His  results 
predicted  selections  comparable  with  other  scaling  techniques,  although 
the  simpler  additive  techniques  were  better  at  predicting  judges* 
decisions.  Klahr  argued  that  this  approach  still  should  be  considered, 
since  the  values  for  worth  were  based  on  subjective  rather  than  nominal 
metrics.  Coombs  (1958)  was  able  to  explain  certain  preferential  incon¬ 
sistencies  by  testing  a  model  based  on  psychological  distance. 

Another  approach  to  worth  assessment  is  functional  measurement 
scaling.  These  methods  involve  monotonic  transformation  of  nominal 
scales  to  subjective  scales  and  then  testing  various  worth  models 
(multiplicative,  additive,  etc.)  using  an  analysis  of  variance.  The 
main  advantage  to  these  methods  is  their  dependence  on  operator  behavior 
rather  than  axioms  positing  ideal  behavioral  patterns  (Anderson,  1970; 
Anderson  and  Shanteau,  1970).  However,  functional  measurement  is  more 
of  a  means  of  scaling  and  testing  various  conjoint  measurement  models 
than  a  separate  measurement  model  in  its  own  right. 


EXPERIMENTAL  COMPARISONS  OF  OPERATOR  AND  TASK  WORTH 
ASSESSMENT  CHARACTERISTICS 

It  is  difficult  to  compare  the  various  measurement  procedures 
empirically,  because  the  criterion  value  is  not  objective.  Also,  better 
performance  for  one  procedure  in  a  particular  experimental  paradigm  does 
not  necessarily  indicate  better  performance  under  different  conditions. 
Even  comparing  procedures'  predicted  results  to  actual  decisions  assumes 
that  the  operator's  decisions  perfectly  reflect  his  preference  structure, 
an  assumption  that  most  of  the  data  contradict. 
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However,  with  these  rather  strong  reservations  in  mind,  empirical 
studies  do  indicate  how  well  the  human  operator  uses  these  procedures 
under  specified  conditions.  In  general,  empirical  studies  show  that 
decomposition  procedures  correlate  well  with  both  outcome  ordering  and 
regression  approaches  to  worth  assessment  (Fischer,  1975). 

Hoepfl  and  Huber  (1970)  compared  two  additive  procedures  with 
subjective  levels  of  the  dimensions  being  derived  by  their  subjects. 

The  criterion  value  was  a  worth  value  assigned  to  outcomes.  For  the 
first  procedure,  subjects  directly  scaled  the  weights  for  each  dimension, 
whereas  for  the  second  approach  the  weights  were  fitted  using  a  least- 
squares  regression  fit.  The  average  correlation  coefficient  for  the 
first  approach  was  0.87  and  for  the  second  0.91  (both  were  corrected  for 
bias).  The  important  finding  was  that  both  models  were  able  to  predict 
the  criterion  value  fairly  well.  Since  the  regression  approach  is  a 
mathematical  minimization  of  the  residual  sum  of  squares,  it  is  not 
surprising  that  it  accounted  for  more  of  the  variance  than  the  scaling 
procedure.  To  compare  the  two  procedures  adequately,  the  regression 
model  would  have  to  have  been  compared  to  criterion  values  other  than 
those  used  to  fit  the  model. 

A  tendency  to  give  more  uniform  weights  for  all  dimensions  was 
noted  for  the  scaling  procedure.  This  tendency  has  been  noted  in  other 
research  on  decomposition  procedures  and  suggests  that  operators  may  be 
conservative  when  scaling  (Edwards,  1978;  Fischer,  1975).  It  appears 
as  if  the  operator  does  not  wish  to  assign  weights  with  too  high  a 
proportion  of  the  total  value  to  any  one  dimension  when  he  decomposes 
an  outcome,  whereas  his  actual  decisions  reflect  heavy  weighting  of 
some  dimensions. 

Huber  and  Daneshgar  (1971)  compared  five  worth  assessment  pro¬ 
cedures  and  found  some  interesting  interactions.  Regression  weights 
were  more  reliable  than  decomposed  scaling  weights  for  experienced 
subjects  in  predicting  job  ratings.  However,  the  opposite  was  true  for 
inexperienced  subjects.  This  suggests  that  decomposing  job  satisfaction 
into  different  dimensions  helps  when  the  subject  is  inexperienced  in 
terms  of  the  outcome,  whereas  experienced  subjects  are  able  to  judRe 
outcomes  (i.e.,  jobs)  directly  more  easily. 

Another  interesting  find  was  that,  in  terms  of  ratings,  a  multi¬ 
plicative  model  was  better  than  an  additive  model,  but,  in  terras  of  the 
job  actually  taken,  the  additive  model  was  a  better  predictor.  It  is 
difficult  to  evaluate  this  because  employment  depends  on  more  than  just 
wanting  a  job,  but  it  does  suggest  that  simpler  models  may  be  better 
predictors  for  situations  other  than  the  one  used  to  generate  the  model. 

Humphreys  and  Humphreys  (1975)  compared  psychological  distance 
models  and  factor  analysis  with  two  additive  decomposition  models  (one 
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with  unit  weight  and  one  with  weights  derived  using  a  gamble-type  pro¬ 
cedure)  .  The  criterion  value  was  how  well  subjects  liked  six  films. 

There  were  six  dimensions  in  the  worth  vector.  Additive  decomposition 
models  were  better  predictors  of  film  choice,  with  the  scaled  weights 
being  better  than  unit  weights. 

Satisficing  reduces  the  cognitive  load  of  the  operator  by  reducing 
the  scope  of  the  multidimensional  worth  vector.  There  is  little  experi¬ 
mental  evidence  to  suggest  that  subjects  use  this  strategy.  Sheridan, 
Richards,  and  Slocum  (1975)  compared  a  satisficing  approach  with  an 
approach  considering  all  dimensions  of  job  worth,  for  nursing  students' 
job  selections.  Their  subjects'  data  did  not  support  a  satisficing 
approach  to  job  selection.  (Jlander  (1975)  had  his  subjects  choose 
either  stocks  (monetary  value)  or  books  (subjective  value),  and  found 
no  evidence  that  satisficing  procedures  were  used  in  the  former  case 
and  only  limited  evidence  in  the  latter  case. 

Another  way  to  reduce  cognitive  load  is  to  choose  outcomes  with  cer¬ 
tain  dimensional  values  rather  than  trying  to  integrate  (or  decompose)  all 
the  information  in  the  worth  vector.  This  would  involve  a  lexicographic 
approach  which  Tversky  (1972  a  and  b)  refers  to  as  "Klimination  by 
Aspect"  (ERA).  EBA  has  two  important  behavioral  consequences:  (1)  an 
outcome's  worth  changes  as  a  function  of  the  value  of  important  dimen¬ 
sions  for  other  outcomes,  and  (2)  operators'  cognitive  loading  is  reduced. 
Tversky  (1972  a)  found  support  for  the  first  hypothesis,  suggesting  that 
his  subjects  used  a  lexicographic  approach. 

For  the  experimental  situations  reviewed,  additive  decomposition 
was  a  relatively  efficient  means  of  assessing  worth.  The  regression 
approach  is  promising,  but  is  rarely  used  to  predict  decisions  in  novel 
situations.  The  use  of  satisficing  and  multidimensional  "distance" 
techniques  received  little  support  from  this  review.  However,  this 
may  be  due  more  to  lack  of  empirical  research  in  this  area  than  actual 
shortcomings  of  these  methods.  Evidence,  both  logical  and  empirical 
(Tversky,  1972  a  and  b) ,  suggests  that  the  operator  uses  lexicographic 
methods  of  worth  assessment  in  certain  situations.  The  latter  finding 
suggests  that,  as  in  probability  estimation,  worth  assessment  may  involve 
heuristic  rather  than  strictly  analytical  processing. 


SUMMARY  OF  WORTH  ASSESSMENT  PROCEDURES 

Table  6  extends  the  task  characteristics  outlined  in  Table  3  by 
Kneppreth  et  al.  The  first  column  lists  task  characteristics.  The 
more  important  measurement  procedures  are  listed  across  the  top  of  the 
table.  Items  A  and  C  are  self-explanatory.  Item  B  is  a  rather  rough 
measure  of  the  difficulty  of  eliciting  worth  values  from  the  operator. 
It  is  given  as  a  range,  since  the  difficulty  varies,  depending  on 
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Che  scaling  procedure  used,  the  number  of  outcomes,  etc.  For  example, 
additive  decomposition  may  be  easier  than  outcome  ordering  if  there 
are  a  large  number  of  outcomes  and  few  dimensions,  while  the  opposite 
is  true  for  few  outcomes  and  many  dimensions. 

If  the  characteristics  listed  in  C  and  D  are  descriptive  of  a 
particular  procedure,  an' X  is  entered  in  the  appropriate  cell.  Since 
not  all  the  task  performance  characteristics  are  self-explanatory,  a 
brief  explanation  is  necessary. 

Analytical  refers  to  the  nature  of  the  worth  assessment  procedure. 

An  analytical  procedure  breaks  down  an  outcome  into  either  a  geometric 
representation  (distance  measured)  or  a  functional  representation  (addi¬ 
tive  or  multiplicative)  of  its  constituent  worth  dimensions.  A  holistic 
task  means  the  value  of  an  outcome  is  judged  without  analyzing  its 
constituent  parts.  Some  procedures  such  as  lexicographic  ordering  and 
satisficing  are  compromises  between  these  two  strategies.  Information 
lost  refers  to  loss  of  worth  information  because  the  worth  vector  is 
not  completely  analyzed. 

Three  levels  of  analysis  of  the  worth  values  elicited  from  the 
operator  are  required.  This  does  not  refer  to  the  worth  assessment 
itself,  but  rather  to  the  analysis  done  of  worth  values  after  elicita¬ 
tion.  For  example,  a  regression  approach  requires  the  operator  only  to 
generate  the  criterion  value,  whereas  rather  sophisticated  regression 
analysis  must  be  done  to  model  the  operator's  worth  vector.  Thus  there 
are  two  issues  concerning  the  difficulty  of  the  worth  assessment  task: 

(1)  how  easy  is  it  to  elicit  a  worth  value  from  the  operator,  and  (2) 
how  much  additional  analysis  is  required  after  obtaining  this  value  to 
determine  the  worth  function?  Sophisticated  analysis  requires  either  an 
expert  in  scaling  (or  statistical)  techniques  and/or  a  computer  program. 

Reduction  in  processing  ioad  indicates  that  the  operator's  processing 
load  is  kept  to  a  minimum  for  a  particular  assessment  problem.  For  dif¬ 
ficult  problems  with  time  constraints,  such  approaches  should  be  con¬ 
sidered;  otherwise  the  worth  assessment  process  might  exceed  the  limited 
processing  capacity  of  the  operator.  Interpretation  difficulty  means 
that  the  worth  function  is  difficult  to  understand  because  the  functional 
form  of  the  underlying  model  is  complex.  Not  widely  used  is  self- 
explanatory. 

Table  6  indicates  that  none  of  the  measurement  procedures  is  superior 
to  the  others.  The  parameters  of  the  worth  assessment  problem  (such  as 
expertise  of  the  assessor,  number  of  outcomes,  time  constraint,  desired 
level  of  precision,  etc.)  determine  the  usefulness  of  the  various  pro¬ 
cedures  for  a  particular  problem. 
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ACTION  SELECTION 


OVERVIEW 

Figure  6  is  an  overview  of  the  action  selection  task.  The  operators’ 
actions  (responses  or  R2)  depend  on  probabilities  P(Hi/d j . . .PCH^/d j ) 
and  outcome  worth  vectors  (<j>x . .  .4>n) .  Thus,  after  processing  incoming 
data  in  terms  of  probability  estimation  and  worth  evaluations  for  possible 
outcomes,  this  information  must  be  integrated  to  choose  an  appropriate 
action.  This  section  will  focus  on  optimal  selection  rules  based  on 
utility  theory. 


ID  ACTIONS  (21  PROBABILITIES  FOR  131  WORTH  VECTORS  FOR 

STATES  OF  THE  WORLD  OUTCOMES 

FIGURE  6.  Decision  Tree  Showing  Action  Selection 
Task  (R^  or  R2) . 

Table  7  lists  the  probability  and  worth  estimations  the  operator 
has  generated. 
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TABLE  7.  Utility  and  Uncertainty 
Information  for  Action  Selection. 


State  of  the  real  world. 


Action 

P 

q 

Enemy 

Friendly 

Sink 

U  (SE) 

U  (SF) 

Not  sink 

U(SE) 

U(SF) 

The  cell  entries  are  utility  (i.e.,  worth)  functions  measuring  the 
worth  vectors  relating  to  the  following  outcomes:  sink  when  enemy  (SE)  , 
sink  when  friendly  (SF),  not  sink  enemy  (SE)  and  not  sink  friendly  (SK) . 
Probabilities  (p  or  q)  of  real-world  states  are  listed  over  the  columns. 
The  operator's  task  is  to  find  rules  for  action  selection,  based  on  the 
type  of  information  listed  in  Table  7,  which  optimizes  some  decision 
criterion. 

This  entails  finding  utility  functions  for  outcomes  when  the 
operator  is  unsure  of  what  outcomes  will  result  from  his  actions.  In 
one  sense,  this  involves  conjoint  worth  measurement  with  probability 
being  one  dimension  of  the  worth  vector.  However,  the  development  ol 
utility  theory  presupposes  an  uncertain  environment  which,  in  itself, 
puts  unique  processing  demands  on  the  operator  (Kahneman  and  Tver sky , 
1978;  Slovic  and  Lichtenstein,  1968). 


UTILITY  THEORY 
Axioms 

Utility  is  a  measure  of  the  worth  of  an  outcome  in  an  uncertain 
environment.  For  simplicity's  sake,  it  will  be  assumed  that  we  are 
comparing  two  outcomes  X  and  Y,  with  X  having  probability  p  and  Y 
Itaving  probability  q  of  occurring.  Six  axioms  are  necessary  to  define 
the  measurement  model  underlying  a  utility  scale  (Coombs,  Dawes  and 
Tversky,  1971).  They  will  be  discussed  very  briefly  with  emphasis  being 
given  to  those  axioms  which  have  significant  behavioral  consequences. 

Utility  theory  involves  bets  of  the  form  (X,  p,  Y)  or  (Y,  q,  X) 
where  q  -  1-p.  This  states  that  X  will  occur  with  probability  p  and  Y 
with  probability  q  (or  1-p).  Expected  utility  is  a  measure  of  the  worth 
of  such  a  bet.  For  example,  how  much  money  are  you  willing  to  pay  for 
rolling  a  die  which  pays  off  X  with  probability  p  and  pays  nothing  Y 
with  probability  q?  The  action  selection  problem  can  be  thought  of  as 
representing  two  bets.  For  action  "sink,"  the  bet  is  of  the  form  (SE, 
p,  SF),  whereas  for  action  "not  sink"  the  bet  is  (SE,  p,  SF) . 
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The  development  of  utility  theory  in  terms  of  bets  most  likely  has 
its  roots  in  the  historical  interest  of  probability  theorist  in  gambling 
situations.  A  U  will  suffix  utility  axioms.  The  first  two  axioms  (U1 
and  U2)  require  the  elements  of  the  bet  to  be  possible  and  the  preference 
structure  of  the  bets  to  be  in  accordance  with  an  ordinal  preference  model. 

U3  requires  that  the  decision-maker  be  indifferent  (~)  between  a 
simple  bet  and  a  compound  bet  having  the  same  expected  outcome. 

If  (X,p,Y)  =  B,  then  (B,q,Y)  ~  (X,pq,Y)  (U3) 

A  compound  bet  is  a  bet  where  the  payoff  is  another  bet.  The 
probability  of  the  operator  obtaining  B  (which  itself  is  a  bet)  is  q. 

If  he  obtains  B,  he  has  probability  p  of  obtaining  X,  thus  the  value 
of  the  compound  bet  must  equal  a  simple  bet  for  which  the  operator 
has  a  pq  probability  of  obtaining  X. 

For  example,  X  is  $50  and  Y  is  nothing.  The  operator  should  not 
be  concerned  with  the  form  of  the  bet;  only  its  expected  outcome.  A 
bet  that  requires  two  successes  in  a  row  in  order  to  obtain  $50  should 
have  the  same  utility  as  a  bet  which  pays  $50  for  one  success,  if  the 
probability  of  a  single  success  is  0.50  in  the  first  case  and  0.25  in 
the  second  case. 

U4  indicates  that  the  utility  of  two  equally  valued  outcomes  (X 
and  Y)  should  not  be  affected  if  they  are  both  involved  in  bets  with 
another  alternative. 


If  X  ~  Y,  then  (X,p,Z)  ~  (Y,p,Z)  (U4) 

U5  states  that,  if  X  is  preferred  to  Y,  then  the  value  of  any  bet 
involving  them  is  less  than  X  and  greater  than  Y. 

If  X  >  Y  then  X  >  (X,p,Y)  >  Y  (U5) 

U6  insures  that  an  outcome  can  be  made  to  be  indifferent  to  a  bet 
that  involves  a  more  preferred  (X)  and  a  less  preferred  outcome  (Z)  as 
alternatives . 

If  X  >  Y  >  Z,  a  p  can  be  found  for  which 

Y  ~  (X,p,Z)  (Ub) 


f 
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The  utility  function  for  the  various  imtcomi’s  can  bi*  derived  I  rom 
U6.  If  X  and  Z  are  chosen  so  that  X  is  the  best  outcome  and  Z  is  the 
worst  outcome,  then  the  p  vaLues  elicited  for  intermediate  outcomes 
would  constitute  a  utility  scale  with  X  having  a  value  of  one  and  Z  a 
value  of  zero  (cl.,  Keeney  and  Raifia,  1976;  Raiffa,  1968). 

In  the  terminology  of  gambling,  actions  with  uncertain  outcomes  can 
be  thought  of  as  a  bet  in  the  form  (X,  p,  Y) .  A  utility  scale  allows 
the  decision-maker  to  compute  the  expected  utility  of  such  a  bet 
(Equation  11). 

E(Ri)  =  p  U(X)  +  q  U (Y)  (11) 

where  U(X)  and  U(Y)  are  utility  functions  defined  on  X  and  Y. 

Tue  optimal  dec ision-maker  chooses  actions  which  maximize  expected 
uLility.  Thus  the  worth  of  actions  in  the  problem  presented  in  Table  7 
can  be  expressed  as: 


E(S)  =  p  tl(SE)  +  q  U(Sl') 
E(NS)  =  p  U (SE)  +  q  U(SF) 


and  the  decision  ruie  would  be  to  choose  the  action  with  the  higher 
expected  utility. 

Other  Dec ision  Rules  Based  on  Utility  Theory 

In  many  real-life  tasks,  the  odds  (or  probabilities)  concerning 
possible  states  of  the  world  change,  but  the  utility  values  for  different 
outcomes  remain  fairly  stable.  In  such  an  environment  the  operator  may 
wish  to  use  a  decision  criterion  based  on  odds.  If  the  odds  are  above 
a  certain  level  action,  A  is  selected  (R/\) ;  whereas  odds  below  the 
criterion  favor  the  selection  of  action  B(Rg) .  Such  a  criterion  is 
referred  to  as  8  (Green  and  Swets,  1966).  The  advantage  of  b  is  that 
the  operator  can  precompute  (or  computerize)  utility  functions  and  make 
decisions  based  only  on  his  subjective  probabilities  (odds)  concerning 
the  changing  state  of  the  world. 

To  derive  such  a  rule,  odds  must  be  found  for  which  the  optimul 
operator  would  be  indifferent  between  R.,  or  R[>.  This  would  be  true 
when  the  expected  utilities  of  the  actions  are  equal. 
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E(Ra)  -  E(Rb) 
or 

p  U(X)  +  q  U(Y)  -  p  U(2)  +  q  U(W) 


p/q  -  (U(W)  -  U(Y))/<U(X)  -  U(Z)) 
or 

indifference  fl  or  0  -  <U(W)  -  U(Y))/(U(X)  -  U(Z))  (12) 

a-b 

Equation  12  indicates  that  such  a  criterion  can  be  developed,  if  the 
operator  can  generate  the  utility  functions  for  possible  outcomes. 


MAUT  involves  a  conjoint  worth  measurement  in  an  uncertain  environ¬ 
ment.  The  utility  of  an  outcome  is  a  function  of  the  decomposed  utility 
vector  with  each  dimension  of  utility  being  called  an  attribute.  A 
scaling  technique  based  on  U6  can  be  used  to  scale  the  multi-attribute 
worth  vector  (see  Kenney  and  Raiffa,  1976;  Raiffa,  1971). 

Like  other  conjoint  models,  the  worth  function  (in  this  case  a 
utility  function)  is  either  a  multiplicative  or  additive  result  of  the 
values  for  each  of  the  attributes.  The  resultant  function  must  be 
commensurate  with  the  utility  axioms,  and  the  utility  function  for 
separate  attributes  must  be  independent  of  the  levels  of  other 
attributes  (Keeney  and  Raiffa,  1976). 

Different  Criteria  for  Action  Selection 

Besides  the  maximum  utility  principle,  there  are  a  number  of  other 
criteria  on  which  action  selection  can  be  based  (Coombs,  Dawes  and 
Tversky,  1971). 

Worth  Selection:  Only  one  outcome  is  considered  possible  for  each 
action;  thus  only  the  worth  of  an  outcome  is  considered. 

Minimax:  Outcomes  assessed  in  terms  of  possible  losses.  The 
chosen  action  is  that  which  results  in  the  least  undesirable  outcome. 

Maxltnln:  Operator  chooses  the  action  for  which  the  worst  possible 
outcome  has  the  hLghest  value. 


Multi-Attribute  Utility  Theory  (MAUT) 
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Maxima x :  Operator  chooses  the  action  for  which  the  best  outcome 
has  the  highest  value. 

Equal  Probability;  Operator  has  Insufficient  reason  to  assign 
probabilities  to  outcomes;  therefore,  he  assumes  all  outcomes  have  the 
same  probability. 

All  these  methods  can  be  criticized,  because  the  long-run  effect 
ol  using  them  would  be  a  loss  of  utility  (Lee,  1971).  However,  they  all 
luve  the  advantage  of  being  relatively  easy  to  use,  since  the  operator 
does  not  have  to  estimate  the  probability.  Some  of  these  techniques 
may  be  especially  useful  for  screening  actions  with  undesirable  outcomes. 


OPERATOR  PERFORMANCE 
Introduction 

Action  selection  is  the  use  of  both  probability  and  worth  informa¬ 
tion  to  select  appropriate  responses  in  terms  of  some  decision  criteria. 
The  expectancy  principle  (expected  utility)  is  the  optimal  solution. 
However,  utility  theory  puts  stringent  constraints  on  the  operator. 

Tlve  first  issue  addressed  will  be  how  well  the  operator's  performance 
conforms  to  these  constraints.  Next,  operator  performance  will  be 
compared  to  other  formal  models  of  action  selection.  Finally,  models 
based  on  the  operator’s  information-processing  abilities  will  be  reviewed. 

Performance  in  JYoms_of  Utility  Axioms 

Because  it  is  easy  to  think  of  utility  axioms  in  terms  of  bets, 
most  of  the  classical  research  on  utility  used  a  gambling  paradigm  to 
test  its  axioms.  An  obvious  inconsistency  between  operator  performance 
and  utility  theory  involves  the  operator  use  of  subjective  probabilities; 
whereas  the  expected  utility  principle  (EU)  assumes  that  the  probabilities 
are  objective.  Edwards  (1955)  and  Savage  (1954)  proposed  that  utility 
theory  be  redefined  to  assume  that  utility  axioms  are  based  on  subjective 
probabilities.  Action  selection,  then,  is  based  on  the  operator's 
attempt  to  maximize  subjective  expected  utility  (SEU)  (Edwards,  19551. 
However,  a  perspective  theory  of  operator  performance  based  on  SEU  is 
at  best  a  doubtful  proposition. 

One  of  the  tenets  of  utility  theory  is  path  independence.  This 
basically  means  that  the  operator  is  not  concerned  with  the  individual 
terms  in  Equation  11,  only  the  result.  This  is  partly  a  consequence 
of  axiom  UJ.  Experimental  research  indicates  that  compound  bets  are 
evaluated  differently  than  simple  bets  directly  contradicting  U3  (Slovic, 
1964).  However,  this  is  probably  due  more  to  the  operator  misaggregat lug 
probabilities  than  to  any  basic  Incompat lb ( l i ty  of  operator  performance 
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and  utility  axioms.  A  more  problematical  finding  for  the  concept  of 
path  independence  is  based  on  research  done  by  Slovic  and  Tver sky 
(1974).  There  are  classical  cases  (developed  by  Allais  and  Ellsberg) 
where  the  operator  chooses  actions  (bets)  that  are  incompatible  with 
the  SEU  principle,  but  are  Justifiable  on  rational  grounds.  One  such 
example  is  illustrated  in  Table  8.  In  Slovic  and  Tversky's  experiment, 
subjects  were  told  to  choose  from  gamble  1  or  2  in  situation  A  and 
gamble  3  or  4  in  situation  B.  A  majority  of  subjects  chose  gamble  1 
and  gamble  4.  However,  no  matter  what  subjective  probability  and 
utility  function  a  subject  uses,  these  choices  are  incompatible  with 
SEU  (Raiffa,  1971).  This  can  be  demonstrated  by  assigning  values  to 
end-points  for  these  two  scales;  for  1.00  on  the  probability  scale  use 
1.00  and  for  no  money  on  the  utility  scale  use  zero.  The  symbols  in 
parentheses  can  then  be  used  to  generalize  the  results  to  any  utility 
or  subjective  probability  function.  According  to  SEU,  preferring 
gamble  1  is  represented  by  the  following  inequality. 


a  >  pe  +  qa 


or 


a(l-q)  >  pe 


However,  preferring  gamble  4  to  gamble  3  implies  the  following  inequality. 


a(l-q)  <  pe 


TABLE  8.  Probabilities  and  Payoffs  for  Allais'  Gambling 
Situation  With  Utilities  and  Subjective 
Probabilities  in  Parentheses. 


Gamble 

Probability 

(pi)  , 

Payoff,  dollars 

(U) 

Situation 

A 

• 

1 

1.00 

(1) 

1  million 

(a) 

2 

.10 

(P) 

5  million 

(e) 

.89 

(q) 

1  million 

(a) 

.01 

(r) 

Nothing 

(0) 

Situation 

B 

3 

1 

.11 

(l-q) 

1  million 

(a) 

.89 

(q) 

Nothing 

(0) 

4 

.10 

(p) 

5  million 

(e) 

.90 

(i-P) 

Nothing 

(0) 

5- 


I 


TP  6124 


Since  these  preference  structures  are  incompatible,  most  subjects 
did  not  choose  according  to  SEU.  What  makes  this  paradigm  important  is 
that,  even  after  explaining  the  reasoning  behind  the  SEU  interpretations, 
most  of  Slovic  and  Tversky's  subjects  still  preferred  1  to  2  and  4  to  3. 
Thus,  it  can  be  argued  that  it  is  not  the  operator's  lack  of  information 
that  is  responsible  for  his  choices,  but  the  fact  that  he  has  different 
criteria  than  SEU  for  action  selection.  Kahneman  and  Tversky  (1978)  use 
similar  examples  to  further  demonstrate  that  the  operator's  preference 
structure  is  not  necessarily  consistent  with  SEU. 

Axiom  U6  can  be  interpreted  as  stating  that  every  bet  has  a  certainty 
equivalent  (or  cash  equivalent).  Thus  for  every  bet  of  the  form  (X,p,Y) 
there  is  some  utility  value  (or  perhaps  money)  which  the  operator  would 
consider  equal  to  the  bet.  Becker,  Groot  and  Marschak  (1964)  manipulated 
the  form  of  the  bet  in  order  to  have  a  number  of  bets  with  the  same  cash 
equivalent.  The  variation  among  their  subject's  cash  equivalents,  in 
situations  for  which  they  should  have  been  equal,  demonstrated  quite 
nicely  that  the  relationship  between  an  uncertain  situation  and  a 
certainty  equivalent  is  not  apparent  to  most  operators. 

Axiom  U4  implies  that  the  context  (i.e.,  values  of  other  outcomes) 
of  the  bet  should  not  change  the  utility  value  of  outcomes.  However, 
experimental  evidence  indicates  that  subjects  consider  the  conLext  of 
the  bet  and  not  simply  isolated  utility  functions  when  making  choices 
(Coombs,  Dawes,  and  Tversky,  1971;  Tversky,  1972a). 

Equation  11  implies  that  the  operator  makes  independent  estimates 
of  probability  and  utility  in  order  to  compute  SEU  (Savage,  1954). 

However,  Slovic  (1966)  gives  evidence  that  the  utility  of  an  outcome 
influences  the  operator's  subjective  probabilities.  Some  subjects 
make  optimistic  probability  estimates  in  light  of  a  big  payoff,  whereas 
others  give  pessimistic  estimates.  Thus  not  only  do  operators'  perform¬ 
ance  characteristics  run  contrary  to  SEU,  but  also  these  characteristics 
appear  dependent  on  personality  differences. 

In  general,  operator  performance  differs  from  utility  axioms  in 
many  important  aspects.  Still  it  is  possible  that  action  selection 
based  on  SEU,  although  theoretically  deficient,  is  a  good  predictor  of 
operator  performance.  The  next  section  will  compare  SEU  to  other 
possible  worth  assessment  models. 

SEU  as  a  Measurement  Model 


SEU  theory  states  that  operators  attempt  to  maximize  average  (or 
long  run)  worth  (or  utility)  by  using  a  selection  model  which  weights 
worth  estimates  with  probability  estimates.  However,  the  operator  may 
view  the  selection  task  in  terms  of  different  processes  than  thosi 
posited  by  utility  theory.  The  simple  bet  paradigm  can  be  represented 
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by  a  vector  with  four  terms  (X,p,Y,q).  SEU  is  one  model  of  integrating 
these  terms  to  assess  the  worth  of  different  bets  (i.e.,  actions). 
However,  empirical  results  indicate  that  the  operator  considers  more 
features  of  the  action  selection  problem  than  implied  by  the  SEU  model 
and  integrates  the  dimensions  differently. 

The  variance,  skewness,  "gambling"  situation  itself  and  type  of 
response  all  influence  the  worth  assessments  of  different  actions 
(Coombs  and  Pruitt,  1955;  Lee,  1971;  Royden  and  Walsh,  1959;  Slovic, 

1972;  and  Tversky,  1967).  Thus  the  operator  considers  more  than  just 
possible  outcomes  and  their  probabilities  when  choosing  actions. 
Furthermore,  even  the  manner  in  which  the  operator  uses  the  worth 
vector  to  assess  the  value  of  particular  actions  differs  from  SEU 
theory.  Different  operators  focus  on  different  dimensions  of  the 
worth  vector,  depending  on  both  personality  differences  and  type  of 
response  (Slovic  and  Lichtenstein,  1968).  Apparently,  the  operator's 
initial  determination  of  worth  is  based  on  the  dimension  (e.g.,  payoff, 
probability  of  loss,  etc.)  he  feels  is  most  important,  Next,  the 
operator  uses  some  method  to  integrate  the  other  information  in  the 
worth  vector  with  the  most  important  dimension  in  order  to  make  a  final 
evaluation  (Slovic  and  Lichtenstein,  1968).  This  suggests  that  no 
formal  measurement  model  could  accurately  describe  the  operator's 
action  selection  process  unless  assumptions  are  made  concerning  the 
operator's  information-processing  capabilities.  Furthermore,  the 
operator  is  limited  in  his  ability  to  resolve  utility  differences 
among  outcomes  (Luce  and  Shipley,  1962;  Suppes  and  Walsh,  1959) . 

It  is  not  clear  exactly  what  process  the  operator  uses  to  integrate 
the  worth  vector.  The  averaging  process  implied  by  SEU  theory  predicts 
operator  performance  in  at  least  an  approximate  sense  (Anderson  and 
Shanteau,  1970;  Tversky,  1967).  However,  the  resulting  value  is  subad¬ 
ditive  (i.e.,  pX  +  qY  <  SEU)  (Shanteau,  1974). 

In  summary,  a  model  that  explains  operator  action  selection  must 
consider  the  following:  (1)  other  factors  besides  outcomes  and  proba¬ 
bilities;  (2)  processing  limitations  of  the  operator;  and  (3)  subadditive 
averaging  process.  Models  which  focus  on  these  issues  will  be  summarized 
in  the  next  section. 

Models  of  Operator  Action  Selection  Performance 

Models  of  human  information  processing  identify  three  serial  pro¬ 
cessing  stages:  stimulus  encoding,  cognitive  processing,  and  response 
selection  (Briggs  and  Swanson,  1969;  Sternberg,  1969).  Cognition 
involves  the  structuring  of  encoded  stimuli  into  meaningful  categories, 
whereas  response  selection  involves  translating  these  stimulus  categories 
into  the  chosen  response.  There  seem  to  be  two  substages  of  cognition 
involved  in  decision-making.  The  first  stage  involves  a  general 
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categorization  and  evaluation  of  possible  actions,  whereas  the  second 
stage  is  concerned  with  the  formation  of  decision  rules  which  result 
in  response  selection  (Slovic  and  Lichtenstein,  1968). 

Kahneman  and  Tversky  (1978)  propose  a  model  of  decision-making 
they  refer  to  as  "prospect"  theory.  A  "prospect"  is  the  worth  and 
probability  values  associated  with  an  action.  After  encoding  these 
values,  the  operator  categorizes  prospects  using  various  processing 
strategies  such  as  recoding,  combining,  and  segregating  different 
dimensions  of  the  prospect.  Next,  an  empirical  function  is  used  to 
weigh  probabilities  (ir(p))  with  small  probabilities  being  "over-valued," 
but  most  of  the  probability  range  is  "under-valued"  compared  to  risk¬ 
less  prospects.  This  process  helps  explain  Allais'  example.  Since 
1.00  would  be  over-valued  in  terms  of  the  probability  scale,  such  a 
weighting  function  could  result  in  the  observed  preferences  of  Slovic 
and  Tversky's  subjects.  Also,  such  a  function  would  explain  subadditivity. 

The  decision  rule  the  operator  uses  for  action  selection  is  similar 
to  the  expectancy  principle,  but  it  focuses  on  the  operator's  potential 
change  in  fortune.  In  simple  bet  cases,  when  X>V>0orX'Y'0, 
Equation  13  is  used  to  determine  the  operator's  evaluation  of  prospects. 

Vk  -  Y  +  u(p)  (X  -  Y)  (13) 


where  Y  is  the  minimum  loss  or  gain  and  Vk  is  the  worth  of  the  kth  action 
(or  prospect). 

Y  is  the  amount  the  operator  gains  or  loses  for  certain,  with  the 
second  term  of  Equation  13  measuring  the  operator's  potential  change  in 
fortune  depending  on  the  result  of  the  action.  This  model  of  decision¬ 
making  is  used  to  explain  empirical  functions  rather  than  to  posit 
ideal  functions,  it  explains  most  of  the  observed  discrepancies  between 
operator  performance  and  SEU  theory  and  serves  as  a  useful  approximation 
of  operator  performance.  It  is  not  as  elegant  as  SEl)  theory,  since  it 
introduces  both  additional  parameters  (n(p))  and  additional  assumptions 
concerning  operator  processing  limitations,  in  order  to  explain  empirical 
results.  Also,  probability  estimation  and  worth  evaluation  are  not 
specifically  addressed  by  the  model.  However,  the  advantage  of  such  a 
model  is  that  it  attempts  to  describe  decision-making  in  terms  of  operator 
performance  characteristics  instead  of  ideal  performance  criteria  which 
may  have  little  empirical  validity. 

Risk-averse  functions  are  a  good  example  of  the  difference  between 
a  normative  and  empirical  approach  to  model  building.  Most  operators 
show  a  tendency  towards  being  risk-averse.  That  is,  they  would  pay  more 
for  a  sure  thing  than  for  an  equivalent  bet  with  the  same  expected  value. 
Pratt  (1964)  and  Keeney  and  Raiffa  (1976)  introduce  a  number  of  utility 
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functions  to  describe  risk-averse  behavior  which  are  commensurate  with 
utility  theory.  However,  Tversky  and  Kahneman  (1978)  point  out  that 
most  empirically  derived  risk  functions  are  not  commensurate  with  utility 
theory.  Their  approach  was  to  fit  the  model  to  the  data  rather  than 
vice  versa. 

Another  model  that  has  both  normative  and  descriptive  appeal  is 
the  "EBA"  model  mentioned  previously.  The  probability  of  choosing  an 
action  is  proportional  to  the  lexicographic  weights  of  different  dimen¬ 
sions  in  the  worth  vector.  Empirically,  the  model  is  supported  by  the 
tendency  of  the  operator  to  focus  on  important  dimensions  rather  than 
to  compute  trade-off  functions  (Slovic  and  Lichtenstein,  1968;  Tversky, 
1972  a  and  b) . 


SUMMARY 

The  maximum  expected  utility  principle  was  shown  to  be  the  normative 
criterion  for  action  selection.  Using  this  criterion,  decision  rules 
were  derived  for  outcomes,  odds  and  multi-attribute  decision  problems. 
Subjective  expected  utility  (SEU)  was  examined  as  a  possible  descriptive 
model  of  operator  performance.  Both  because  of  serious  violations  of 
axioms  and  processing  limitations  of  the  operator,  empirically  derived 
models  such  as  "prospect"  theory  were  found  to  be  superior  to  a  model 
based  on  oEU  theory. 

Table  9  summarizes  the  performance  aspects  of  different  action 
selection  models. 
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INFORMATION  SELECTION 


OVERVIEW 

The  iterative  nature  of  information  selection  can  be  seen  by 
examining  the  feedback  loop  in  Figure  2.  Before  selecting  an  action, 
the  operator  has  the  option  of  sampling  more  data,  depending  on  both 
his  own  state  of  uncertainty  and  the  cost  of  sampling.  Cost  is  used 
in  a  general  sense  indicating  cost  in  terras  of  physical  and  temporal 
constraints  as  well  as  monetary  value. 

The  internal  dynamics  of  information  selection  are  illustrated  in 
Figure  7.  Information  selection  involves  decisions  concerning  both 
"how  much"  and  "what  type"  of  information  to  select  (Nickerson  and 
Feehrer,  1974).  The  latter  process  is  designated  as  information 
gathering  (Box  A  in  Figure  7).  This  involves  a  selection  among  various 
information  sources  as  a  function  of  average  information  (Ij)  and 
cost  of  sampling  (Cj)  for  each  source.  Formally,  information  is  a 
measure  of  the  reduction  in  operator  uncertainty  after  sampling  a 
datum  (Equation  14)  and  average  information  is  the  expected  informa¬ 
tional  value  of  data  from  a  particular  source  before  sampling. 

Ij  “  -log  P(dj|Hk)  U*) 

where  Hk  is  the  hypothesis  reflecting  the  actual  state  of  the  world. 

In  terms  of  our  example,  information  gathering  would  consist  of 
deciding  which  sensor  devices  to  use  and  whether  to  get  an  updated 
intelligence  report  from  an  E-2  (tracker  aircraft)  or  strike  command. 
Information  in  its  technical  sense  would  be  measured  in  terms  of  the 
increase  in  probability  for  the  true  hypothesis  that  each  information 
source  would  result  in.  Cost  would  depend  on  such  factors  as  time  to 
retrieve  information,  danger  involved  in  breaking  radio  silence,  etc. 

Since  Equation  14  is  simply  a  transformation  of  probability, 
information  can  be  Interpreted  in  terms  of  Bayesian  statistics 
(Analytics,  1976;  Savage,  1954).  For  example,  diagnost ici ty ,  in 
Bayesian  terms,  is  simply  the  algebraic  difference  between  the 
informational  value  of  the  denominator  and  the  informational  value 
of  the  numerator  of  the  likelihood  ratio.  There  are  a  number  of 
advantages  in  using  information  metric  to  measure  operator  performance. 
The  metric  is  independent  of  the  type  of  data  selected  and  experimental 
paradigm  used.  Also,  this  metric  has  been  used  to  measure  human  per¬ 
formance  for  a  wide  variety  of  cognitive  and  motor  tasks,  allowing 
operator  performance  in  information  selection  to  be  compared  to  other 
human  performance  results  (Briggs,  1974;  Fitts,  1954;  Hicks,  1951). 
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The  second  process  involved  in  information  selection  is  optional 
stopping  (Box  B).  The  "fair  cost"  of  sampling  a  data  source  is  the 
difference  in  value  of  selecting  an  action  before  and  after  sampling 
(Wendt,  1969).  The  optional  stopping  task  involves  deciding  whether 
the  "fair  cost"  of  sampling  the  favored  data  source  is  more  or  less 
than  the  actual  cost  of  sampling.  If  it  is  more,  then  sampling  takes 
place,  otherwise  the  operator  stops  sampling  and  selects  the  action 
with  the  highest  expected  utility 

The  problem  our  exemplary  pilot  would  have  concerns  his  trade-off 
between  uncertainty  reduction  and  danger  to  his  aircraft.  The  closer  he 
gets  to  the  ship  (and  thus  the  more  information  he  samples)  the  more 
likely  the  ship  is  to  fire  antiaircraft  missiles  at  him.  Also,  of  course, 
the  more  likely  he  would  be  to  identify  the  ship.  The  optional  stopping 
task  consists  of  deciding  when  to  stop  gathering  more  information  and 
either  launch  the  missile  or  stop  flying  toward  the  ship  and  continue 
on  his  original  flight  plan. 

Thus  information  selection  involves  both  the  first  (A)  and  last 
(B)  thing  an  operator  does  before  committing  himself  to  a  course  of 
action  (i.e.,  (a)  he  must  decide  which  data  to  sample,  and  (b)  he 
must  decide  when  to  stop  sampling  and  start  acting). 


EMPIRICAL  RESULTS 
Overview 


Most  of  the  experimental  paradigms  reviewed  were  similar  to  those 
used  for  the  Bayesian  updating  tasks.  The  main  difference  consisted 
in  the  dependent  variables  which  are  number  of  data  sampled  and  their 
informational  value  for  the  present  task. 

In  general,  subjects  sample  both  too  much  and  too  little  i. formation 
depending  on  the  experimental  parameters  (Nickerson  and  Feehrer,  1976). 
When  the  information  value  of  the  data  source  is  too  low,  too  much 
sampling  cakes  place;  whereas  too  little  sampling  takes  place  for 
information  sources  with  relatively  high  values  (Nickerson  and  Feehrer, 
1976;  Snapper  and  Peterson,  1971).  There  seem  to  be  individual  dif¬ 
ferences  among  operators,  with  some  being  overly  cautious  (want  too 
much  information)  and  others  too  incautious  (sample  too  little  informa¬ 
tion)  (Edwards  and  Slovic,  1965).  Also,  there  seems  to  be  a  nonoptimal 
relationship  between  the  amount  of  information  sampled  and  the  amount 
available.  Apparently,  the  operator  is  conservative.  He  does  not  want 
to  deplete  his  information  source  so  he  weights  his  estimate  of  the 
"fair  cost"  of  sampling  as  a  function  of  the  sample  size  available 
(Levine,  Samet  and  Brahlek,  1975;  Pitz,  1968). 
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The  operator  is  able  to  use  estimates  of  prior  odds,  diagnosticity 
and  cost  to  determine  when  optional  stopping  should  take  place,  but  his 
actual  criteria  are  not  optimal  (Wendt,  1969).  The  operator  is  able  to 
learn  how  to  be  more  efficient  in  an  information  gathering  task,  but 
performance  is  still  less  than  optimal  after  practice  (Levine  and  Samet, 
1973).  Optional  stopping  seems  to  be  more  time-consuming  than  informa¬ 
tion  gathering  (Levine  and  Samet,  1973). 

An  experiment  by  Pitz  (1968)  shows  the  relationship  of  operator 
performance  for  optional  stopping  and  Bayesian  updating.  His  subjects 
sampled  a  limited  number  of  red  and  blue  poker  chips  from  two  bags. 

This  was  similar  in  design  to  the  Urn  problem  in  that  the  probability 
of  a  sample  being  from  a  particular  bag  was  a  function  of  the  difference 
(d*)  in  the  number  of  red  and  blue  chips  (the  Urn  example  is  reviewed 
in  the  probability  estimation  section).  If  cost  were  held  constant, 
the  criterion  for  optional  stopping  should  be  based  on  d*.  However, 
subjects  used  a  criterion  more  sensitive  to  the  proportion  of  red  and 
blue  data  samples  than  to  d*.  This  directly  verifies  the  previously 
reviewed  results  of  Marks  and  Clarkson  (1973)  which  indicated  that 
subjects  estimated  probability  by  proportion  of  different  colored 
chips  and  not  by  the  Bayesian  solution  based  on  d*. 

The  point  of  the  comparison  is  that  the  results  should  be  the 
same.  The  processing  requirements  for  estimation  of  informational 
value  and  probability  are  identical  in  many  respects.  Therefore, 
when  the  other  parameters  of  the  situation  are  similar,  we  should 
expect  the  operators  decision-making  characteristics  to  be  similar 
for  these  two  tasks.  This  suggests  that  heuristics  used  for  probability 
estimation  are  most  likely  evinced  by  the  operator  in  information 
selection  tasks  also* 

Time  Constraints 


It  was  mentioned  previously  that  time  constraints  could  be  con¬ 
sidered  as  a  cost  factor,  Pachella  and  Pew  (1968)  give  empirical 
evidence  that  the  operator  uses  both  time  and  monetary  cost  criteria 
for  optional  stopping  in  a  choice  reaction  time  task.  One  of  the  more 
interesting  findings  concerning  the  operator  decision  to  stop  sampling 
and  start  findings  concerning  the  operator  decision  to  stop  sampling 
and  start  responding  is  the  so-called  "speed-accuracy"  trade-off  (Fitts, 
1955;  Pew  1969).  This  trade-off  implies  that  the  operator  is  able  to 
sample  information  at  a  constant  rate,  but  chooses  different  "informa¬ 
tional"  or  "time"  criteria  depending  on  whether  the  task  pays  off  for 
speed  or  accuracy.  Apparently,  when  the  operator  is  told  to  "speed  up," 
he  samples  information  at  the  same  rate,  but  samples  for  a  shorter  time 
segment.  The  result  of  such  a  trade-off  is  that  operator  performance 
will  be  more  error-prone  in  circumstances  where  he  speeds  up  because 
he  can  only  sample  a  given  amount  of  information  in  a  particular  time 
segment.  In  the  most  extreme  case,  when  he  is  under  great  time  pres¬ 
sure,  the  operator  may  not  sample  information  at  all,  but  respond 
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randomly  (t.e.,  guess)  (Swensson  and  Edwards,  1971;  Yelloit,  1971). 

If  the  "speed-accuracy  operating  characteristics"  for  particular 
operators  are  known,  then  error  rate  can  be  predicted  for  a  task  that 
has  a  temporal  deadline  attached  to  it  (Pew,  1969). 

Although  the  parameters  of  the  "speed-accuracy"  trade-off  have 
been  investigated  using  different  choice  reaction  time  paradigms,  the 
real-world  validity  of  such  paradigms  is  suspect.  Research  needs  to 
be  done  to  determine  the  "speed-accuracy  operating  characteristics" 
for  different  operators  using  real-world  scenarios  when  such  factors 
as  work  load,  stress,  and  payoff  schedules  are  varied  concomitantly. 


SUMMARY:  INFORMATION  SELECTION 

Information  selection  depends  on  criterion  based  on  both  cost 
evaluation  (worth  assessment)  and  uncertainty  evaluation  (probability 
estimation).  Therefore,  the  operator's  information  selection  character¬ 
istics  depends  on  his  decision-making  characteristics  for  the  other 
stages  of  decision-making.  Heuristics  and  biases  found  in  these  stages 
are  also  pertinent  to  information  selection  performance. 

However,  there  are  a  number  of  performance  characteristics  specific 
to  information  selection: 

1.  When  there  are  a  great  number  of  information  sources  to  choose 
from,  operator  performance  is  degraded.  There  seem  to  be  two  reasons 
for  this:  (a)  the  operator  pays  more  than  the  fair  cost  for  information, 
and  (b)  too  much  information  overloads  the  operator's  processing  capacity. 

2.  Too  sparse  informational  sources  results  in  degraded  performance 
for  the  opposite  reason:  (a)  the  operator  will  not  pay  for  sources  even 
when  their  cost  is  less  than  the  fair  cost,  and  (b)  the  operator  tends 

to  conserve  sources  rather  than  purchase  needed  information. 

3.  The  operator  can  trade  off  the  cost  associated  with  time  for 
Information  depending  on  the  cost  parameters  for  each. 
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ISSUES  AND  TENTATIVE  CONCLUSIONS 


OPERATOR  AND  TASK  CHARACTERISTICS 

This  paper  has  reviewed  operator  and  tasks  performance  character¬ 
istics  for  four  stages  of  decision-making:  information  selection, 
probability  estimation,  worth  assessment,  and  action  selection.  The 
most  important  general  finding  is  that  the  operator  is  best  characterized 
in  terms  of  "bounded  rationality"  rather  than  in  terms  of  normative  or 
optimal  models.  It  is  necessary  to  understand  the  operator's  processing 
limitations  in  order  to  understand  how  he  performs  as  a  decision-maker 
in  diverse  environments. 

The  concept  of  "heuristics"  borrowed  from  the  discipline  of 
artificial  intelligence  (Simon,  1970)  seems  to  be  the  most  promising 
description  oC  operator  decision-making  performance.  A  heuristic  is  an 
algorithm  or  rule  which  attempts  to  find  satisfactory  solutions  without 
analytically  examining  all  possible  solutions.  Heuristic  techniques 
are  used  because  of  the  processing  limitations  inherent  in  a  particular 
decision  problem.  It  is  an  apt  description  of  operator  performance  for 
a  variety  of  reasons: 

1.  Heuristic  rules  lead  to  solutions  which  are  rational  but  not 
necessarily  optimal. 

2.  The  reason  a  heuristic  is  used  has  to  do  with  the  limitations 
of  the  processor. 

3.  Different  heuristics  are  used  for  different  tasks. 

4.  Heuristics  lead  to  predictable  errors. 

The  operator  is  constrained  both  by  his  processing  limitations  and 
his  misunderstanding  of  normative  processes.  A  number  of  biases  and 
performance  characteristics  have  been  outlined  in  this  paper,  and  rather 
than  list  these  biases  again,  the  following  performance  characteristics 
seem  to  best  summarize  the  results: 

1,  Operators  are  not  "conservative"  or  "excessive"  estimators  of 
probability.  Rather,  their  particular  performance  characteristics 
depend  on  the  parameters  of  the  estimation  tasks. 

2.  Operator  biases  are  a  result  of  adaptive  rather  than  irrational 
processes.  They  are  an  attempt  to  make  sense  of  a  noisy  environment. 
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3.  Since  some  of  these  biases  are  predictable  and  fairly  con¬ 
sistent,  it  is  possible  to  use  corrective  functions  that  result  in 
nearly  optimal  predictions. 

4.  Given  sufficient  time,  the  operator  is  able  to  decompose 
outcomes  and  develop  trade-off  functions  which  weigh  the  effects  of  a 
number  of  different  dimensions  in  order  to  compute  the  worth  for  an 
out  come . 

5.  However,  when  the  number  of  dimensions  is  large  or  time  is 
constrained,  the  operator  chooses  assessment  techniques  commensurate 
with  his  processing  limitations  (c.g.,  lexicographic,  satisficing...) 

6.  Action  selection  models  which  are  based  on  empirical  functions 
such  as  Prospect  theory  are  better  predictors  of  performance  than  those 
which  posit  mainly  normative  functions  such  as  utility  theory. 

7.  The  operator  performs  best  in  information  selection  tasks  when 
there  Is  a  moderate  amount  of  Information  to  select  from.  Too  much  or 
too  little  information  results  in  degraded  performance. 

IMPROVING  PKRKORMANCi:  THROUGH  TRAINING  AND  AIDING 

The  paper  has  identified  a  number  of  operator  performance  defi¬ 
ciencies.  Training  and  aiding  are  two  possible  ways  tv'  improve  operator 
performance..  Techniques  tor  improving  probability  estimation  and  worth 
evaluation  (e.g.,  boot st rapping)  performance  have  been  discussed,  this 
section  will  discuss  more  global  approaches  to  performance  enhancement 
in  decision-making. 

Saleh  et  al  (IV78)  have  developed  a  decision-making  program  imbedded 
in  the  anti-submarine  warfare  (ASW)  context  which  trains  the  operator  in 
all  aspects  of  decision-making.  The  import  ant  decisions  the  operator 
must  make  in  this  environment  are  pinpointed  and  defined  as  generic 
decision  tasks  (e.g.,  action  selection,  probability  estimation...). 

Then,  the  operator  Is  trained  to  use  decision  analytic  techniques  tv' 
make  decisions  for  these  important  decision  points  from  the  ASW  environ¬ 
ment.  The  premise  of  this  experimental  program  is  that  the  operator  is 
being  trained  to  use  powerful  decision  analytic  tools  (decision  trees, 
probability  estimations...)  while  at  the  same  time  he  is  trained  tv' 
solve  specific  decision  problems  in  the  ASW  environment.  Since  this 
project  is  still  In  its  early  stages,  it  is  too  early  to  evaluate  its 
success . 
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Crooks,  Kuppin  and  Freedy  (1977)  have  introduced  the  concept  of 
decision-aiding  Into  the  training  domain.  They  trained  troubleshooters 
in  fault-finding  for  electronic  circuits.  The  aiding  device  gave  feed¬ 
back  and/or  aiding  during  the  task.  Feedback  consisted  of  telling  the 
subject  what  the  expert  would  have  done  after  the  decision  was  made. 
Whereas,  aiding  involved  telling  the  subject  what  alternatives  the 
expert  would  have  considered  before  the  decision  was  made.  A  combina¬ 
tion  of  aiding  and  feedback  trained  the  operator  in  making  less  costly 
decisions  even  in  situations  where  no  aiding  or  feedback  was  available. 

However,  in  general,  training  using  decision  analytic  techniques 
or  aids  is  still  in  the  exploratory  stages.  The  advantage  to  such 
training  seems  obvious,  but  it  remains  to  be  seen  whether  the  results 
live  up  to  its  promise.  In  a  similar  vein,  a  number  of  decision  aids 
are  being  developed  to  help  the  operator  in  operational  settings, 
particularly  in  the  command  and  control  environment.  Preliminary 
results  indicate  that  devices  which  result  in  action  selection  are 
not  as  practical  as  devices  which  give  the  operator  information  to  aid 
him  in  making  the  choice.  The  former  devices  tell  rather  than  inform 
the  operator,  leaving  him  uncertain  as  to  the  logic  of  the  chosen 
decision. 

Two  examples  of  the  "information"  type  device  are  an  Emission 
Control  (ECOM)  decision  aid  (Decision-Science,  1978)  and  a  device  to 
aid  in  selecting  optimal  air  strike  routes  (Integrated  Science,  1978). 

In  both  cases,  the  operator  chooses  different  feasible  alternatives, 
sensor  configurations  in  one  case  and  air  strike  routes  in  the  other, 
and  the  aids  display  the  important  parameters  for  each  choice  and 
compute  their  expected  utility  using  criteria  the  operator  is  familiar 
with.  This  allows  tilt*  operator  to  explore  different  alternatives  he  lias 
generated  himself  and  allows  the  computer  to  use  its  computational 
power  to  enhance  the  operator’s  decision-making  performance. 

In  summary,  both  aiding  and  training  seem  promising  as  a  way  to 
enhance  the  operator  decision-making  skills.  The  approaches  are 
complementary,  since  aids  can  improve  training  and  training  is  neces¬ 
sary  for  the  operator  if  he  is  to  appreciate  the  advantages  of  aids. 

Both  approaches  are  still  in  the  developmental  stage.  If  they  are  to 
prove  useful,  their  purpose  will  be  to  enhance  rather  than  to  replace 
the  operator  as  a  decision-maker. 


RELATED  ISSUES  AND  FUTURE  TRENDS 

Although  hypothesis  and  outcome  generation  was  discussed  in  this 
report,  very  little  was  said  concerning  their  performance  characteristic 
This  is  not  meant  to  suggest  that  these  processes  are  unimportant,  only 
that  not  much  is  known  concerning  operator  performance  for  these 
processes.  Preliminary  evidence  suggests  that  the  operator  considers 
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far  too  few  hypotheses  (Gettys,  Fisher,  and  Mehle,  1978)  and  outcomes 
when  structuring  the  decision  problem.  The  problem  of  structuring  the 
decision  situation  may  prove  to  be  the  most  important  component  of 
decision-making  and  should  prove  to  be  a  fertile  ground  for  future 
research  (Saleh  et  al,  1978). 

Most  of  the  aiding  and  training  programs  mentioned  have  their  impetus 
from  a  new  applied  science  referred  to  as  "Decision  Analysis."  Decision 
analysis  "merges  the  logical  foundations  of  statistical  decision  theory 
with  the  capabilities. . .developed  in  the  fields  of  system  analysis  and 
operations  research"  (Matheson,  1969) .  Its  main  impact  has  been  on  the 
business  community  (Brown,  Kahr,  and  Peterson,  1974),  but  the  techniques 
developed  by  decision  analysts  are  widely  applicable  and  have  been  used 
in  many  military  situations.  The  major  drawback  to  decision  analysis 
is  that  it  demands  a  high  level  of  expertise  and  therefore  a  formal 
decision  analysis  should  be  reserved  for  projects  with  a  high  potential 
payoff.  However,  the  technology  of  aiding  and  training  decision-makers 
will  continue  to  benefit  from  the  research  results  generated  by  this 
discipline. 

Artificial  intelligence  (AI)  is  another  discipline  which  has  a 
potentially  important  impact  on  decision-making.  Al  involves  problem 
solving  algorithms  (i.e.,  techniques)  for  situations  where  the  solution 
demands  complex  logical  decisions  such  as  chess,  geometric  recognition, 
etc.  In  other  words,  its  domain  is  problems  previously  thought  to  be 
solvable  only  through  human  intelligence. 

One  of  the  more  applied  uses  of  AI  is  knowledge  engineering 
(Figenbaum,  1978;  Shortliffe  and  Buchanan,  1975).  Basically,  this  is 
the  logical  structuring  of  rules  elicited  from  experts  in  a  particular 
field  to  model  the  experts'  performance  in  some  decision  task.  Computer 
programs  using  knowledge  based  on  algorithms  have  been  successfully  used 
for  disease  diagnosis,  chemical  analysis,  and  in  military  situations  to 
predict  the  presence  of  possible  hostile  forces.  Knowledge  based 
algorithms  are  currently  being  used  for  aiding  in  only  a  few  military 
situations.  However,  the  potential  applications  for  military  problems 
are  great,  especially  with  the  rapidly  increasing  technology  of  both 
computer  hardware  and  software. 
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SEA-03416  (1) 

SKA-09G32  (2) 

1  Chlaf  of  Naval  Research,  Arlington 
1  Buraau  of  Madlclna  &  Surgary  (Coda  513) 

1  Naval  Aaroapaca  Madlcal  Raaaarch  Laboratory,  Panaacola  (Coda  L5) 

4  Naval  Air  Davalopmant  Cantar,  Warnlnatar 

Coda  602  (1) 

Coda  6021  (1) 

Coda  6022  (1) 

Tachnlcal  Library  (1) 

1  Naval  Air  Taat  Cantar  (CT-176),  Patuxent  Rivar  (SY-72) 

1  Naval  Poatgraduata  School,  Montaray  (Tachnlcal  Library) 

1  Naval  Training  Bqulpaant  Cantar,  Orlando  (Tachnlcal  Library) 

2  Navy  Paraonnal  Raaaarch  and  Development  Cantar,  San  Dlago 

Coda  02  (1) 

Coda  03  (1) 

1  Of flea  of  Naval  Raaaarch  Branch  Off lea,  Paaadana 

2  Pacific  Mlaalla  Taat  Cantor,  Point  Mugu 

Coda  1226  (1) 

Tachnlcal  Library  (1) 

1  Amy  Hunan  Engineering  Laboratory,  Aberdeen  Proving  Ground 
1  Aaroapaca  Madlcal  Raaaarch  Laboratory,  Wrlght-Patterson  Air  Porca 
_aae  (Coda  HBA) 

12  Dafanaa  Tachnlcal  Information  Cantar 
1  Dafanaa  Intalllganca  Agency 

1  Anas  Raaaarch  Canter  (NASA)  (Aviation  Safety  Raaaarch  Office, 
Tachnlcal  Library) 

1  Applied  Phyalce  Laboratory,  JHU,  Laurel,  MD 

1  Hughes  Aircraft  Coapany,  Culver  City,  CA  (Dlaplay  System  Laboratory) 
1  Bman  Factors  Raaaarch,  Incorporated,  Golata,  CA 

1  Institute  for  Dafanaa  Analyses,  Arlington,  VA  (Technical  Library) 

2  Me Donnell  Douglas  Corporation,  St.  Louis,  NO  (Engineering  Psychology) 
1  Pareeptronics,  Woodland  Mills,  CA  (Tachnlcal  Library) 

1  The  Boeing  Coapany,  Seattle,  WA  (Craw  System  NS-41-44) 

1  The  Rand  Corporation,  Santa  Monica,  CA  (Library) 


